Spinning up a server is easy enough, just go to Digital Ocean and push a button. But now you are responsible for your server. What does that mean?

Once you have a server running, you need to take care of a few things:

1. Data and databases
2. Monitoring
3. Security
4. Maintenance

Data and databases

Cloud servers and automation tools like Ansible make it very easy to set up a server. If the server crashes, we can just build another one in a few minutes. The important thing is protecting the data. Managing that data and making sure it doesn't get lost or stolen is the biggest driver for hosting.

We typically have two kinds of data, uploaded files and databases. In the old days, we would keep data files on the local hard disk, and either install a database locally or on a dedicated database server.

Object storage

These days, it's common to put uploaded files in cloud "object" storage like Amazon S3 or Digital Ocean Spaces. These services are typically very reliable, and we can be comfortable that if we have written the data, it won't be lost in a hard disk crash.

Backups

There is a saying, "The one way to guarantee that you will get fired as a systems administrator is to fail to make backups."

While writing to an object store can generally be considered reliable, we still need to be careful that a program bug doesn't accidentally delete everything. DevOps tools make it almost as easy to delete an S3 bucket as to create it.

It's useful to make periodic snapshots. If you are in AWS, it's probably sufficient to periodically sync data to another S3 bucket, perhaps in a different region. Another option is to sync data to local disk partition and then back it up to Tarsnap or rsync.net.

An important part of backups is testing that they work. It's sadly common to think that you have been making daily backups, then find that they are bad when you need them.

Security

Make sure that the permissions on your S3 buckets are locked down. It's easy to accidentally make them world writable. Enable encryption, so if someone gets access to the data, they still can't read it.

There is a saying that, "Backups are a way of reliably violating file system access permissions at a distance." Make sure that your backups are as secure as your online systems.

Databases

Running a database on the app server machine is easy. For simple applications where the data doesn't change often, you can set up a cron job to make a daily export of the data and back it up to file storage, as described above. You risk losing up to a day of data written since the last backup.

It's generally better to separate the app server from the db server. It makes it easier to upgrade the app server without worrying about the data and provides a bit more security.

If you need higher availability, the traditional solution is to use replication to synchronize the data from your primary database to a backup database. If the primary fails, you promote the backup to primary and point the application to it. You may lose a transaction or two that is in flight, but generally this is quite safe.

Cloud hosting environments like AWS and GCP have database hosting services that take care of the database for you. If you are running in the cloud, that's what you should use. You can run a single database and make periodic snapshot backups, similar to the export process described above. You can also set up a hot backup for high availability. This typically uses the magic of the cloud service's block storage system to replicate data instead of the database native protocol, so it is less complex and performs better.

Monitoring and alerting

You need some way to make sure your server is working other than waiting for your customers to call. You also need to get notified when it is struggling, e.g. having application errors.

Monitoring

There are two sides to monitoring: external and internal.

External monitoring periodically checks that your service is up, e.g. by making an HTTP request to your home page or (better) a special health check URL.

Internal monitoring checks things that are not visible from outside, e.g. disk space usage.

Logging

Your application and other services on the server will generate logs. You need to monitor them for cries for help.

Alerting

When something goes wrong, your monitoring system should notify you. This might be email, SMS, or a post to a Slack group.

Solutions

While you can set up your own home-grown monitoring system, it's easier to use a service like Datadog. In AWS, you can use AWS CloudWatch.

Security

When you put a server on the internet, people will immediately start attacking it. I have seen machines hacked within five minutes because someone set the root password to abc123.

Minimizing surface area

Minimize the amount of software that you are running. Do things in a simple way. If you are not running a piece of software, you don't have to worry about bugs in that software being exploited to hack your system.

Run a firewall that restricts access to the system. For a straightforward application, that means opening port 22 (or an alternative port) for ssh and port 80/443 for web. You might run a mail server like postfix to facilitate sending mail (though a service like AWS SES or Sendgrid would probably be better). Block inbound mail, and you don't have to worry about the server being hacked.

Tighten up defaults

Only allow remote access with ssh keys, not passwords. Don't allow remote logins as root. Only allow ssh access for your own user account, not that of the application. Use strong passwords everywhere. Use a password manager like LastPass to manage them.

Keep up to date

It's tempting to say, "If it ain't broke, don't fix it." Security vulnerabilities are found all the time, though, so it's important to keep your system up to date, or you will be caught by some script kiddie scanning for known vulnerabilities. Apply updates periodically. The cloud makes this easier than it used to be. Run a stable version of Linux, e.g. CentOS or Ubuntu LTS, and you can safely update packages. For major OS updates, spin up a new instance to try it out. This is where having your db separate from your app server makes things easier.

Run separate dev, staging and prod environments

The cloud makes it easy to make new instances. Keep each environment separate, simple and consistent. That way you can keep production data safe.

Avoid vulnerable software

Keep your custom app separate from third party code, or lock it down as much as you can.

Some software is common and easy to hack. WordPress is the biggest example. It is open source, which means that attackers can inspect it to find vulnerabilities. There are lots of free, poorly written plugins. Millions of people run WordPress, so it's easy to scan the internet looking for vulnerable servers.

If you need to run WordPress, use a managed hosting service, e.g. Bluehost. If you just need a simple blog, look at static site generators, e.g. Jekyll or Pelican. Run your blog out of an S3 bucket and there is nothing to hack or to fail.

Principle of least privilege

There are lots of other things that you can do to lock down your app.

Maintenance

If you have the above done right, there is not much that you need to do to keep your server running. The main thing is paying attention to disks filling up, often from log files. Set up log file rotation, e.g. with logrotate.

Don't worry

This may seem a bit overwhelming, but it's not hard once you get the hang of it.

It's like learning to fix your own car. It may not make sense to change your oil every time, but learn how things work. You won't get stuck on the side of the road with a flat tire or ripped off by mechanics.

The Law of Leaky Abstractions says that we need to understand the layers below our application. If you want to be an architect, you need to understand the laws of physics for hosting. Learning how things work allows you to build better applications and save lots of money.

You are now on your way through the Five Stages of Hosting. I got to stage 4.5, running my own VPS and dedicated server hosting business, before I ascended to the cloud. :-)