Deploying an Elixir app to Digital Ocean with mix_deploy

By Jake Morrison in DevOps on Sat 01 June 2019

This is a gentle introduction to getting your Elixir / Phoenix app up and running on a server at Digital Ocean (affiliate link). It starts from zero, assuming minimal experience with servers.

We build build and deploy to the same server, using Erlang releases to run the code under systemd. It uses the mix_deploy library to handle deployment tasks.

You can run everything (build, web and db) fine on Digital Ocean's smallest $5/month plan, but this guide uses their managed databases service so you don't need to manage the database.

We will be using a boilerplate Phoenix project with PostgreSQL database. It assumes you are running macOS on your dev machine and Ubuntu 18.04 on the server.

These instructions are based on this working example application and the principles described in the blog post "Best practices for deploying Elixir apps".

This post includes the basic instructions on how to prepare your existing Elixir/Phoenix application for deployment using mix_deploy. For more detailed information, please consult the README of the mix-deploy-example repo.

If you have any questions, open an issue on GitHub or ping me on the #elixir-lang IRC channel on Freenode, I am reachfh.

Overall approach

  1. Create the server
  2. Configure ssh
  3. Configure the build / deploy user
  4. Check out code on the server from git and build a release
  5. Deploy the release

It is recommended to first get the template running, then prepare your own project for deployment.

NOTE: This guide works with CentOS 7, Ubuntu 16.04, Ubuntu 18.04 and Debian 9.4. If you are not sure which distro to use, choose Ubuntu 18.04. The approach here works for dedicated servers and cloud instances as well.

The actual work of building and deploying releases is handled by simple shell scripts which you run on the build server or from your dev machine via ssh, e.g.:

ssh -A deploy@web-server
cd build/mix-deploy-example
git pull

# Build release
bin/build

# Extract release to target directory on local machine, creating current symlink
bin/deploy-release

# Run database migrations
bin/deploy-migrate

# Restart the systemd unit
sudo bin/deploy-restart

Create the server

Go to Digital Ocean (affiliate link) and create a Droplet (virtual server).

  • Choose an image: Choose Ubuntu 18.04
  • Choose a plan: Standard is fine
  • Choose a size: The smallest, $5/month Droplet is fine
  • Choose a datacenter region: Select a data center near you
  • Add your SSH keys: Select the "New SSH Key" button, and paste the contents of your ~/.ssh/id_rsa.pub file. Create an ssh key, if you don't have one already. On Mac OS, you can copy your SSH key to clipboard using cat ~/.ssh/id_rsa.pub | pbcopy.
  • Choose a hostname: The default name is fine, but awkward to remember and type. Use "web-server" or whatever you like

The defaults for everything else are fine. Click the "Create" button.

Configure ssh to talk to your server

Note the IP address of your new droplet in the Digital Ocean UI.

Configure ~/.ssh/config on your local dev machine so you can connect to the server.

# Change the IP address below to the actual IP address of your Droplet
Host web-server
  HostName 123.45.67.89

Create a deploy user on the web server

For security, we use two operating system user accounts, the deploy user to build and deploy the app, and the app user to run the app.

Connect to the web server as root:

ssh root@web-server

Create the deploy user:

useradd -m -s /bin/bash deploy

Configure sudo to allow the deploy user run commands as root without a password:

echo "deploy ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/10-app-deploy

There are more sophisticated ways to manage users. We normally manage user accounts with Ansible.

Configure ssh access to the deploy user.

Create the .ssh directory and set permissions:

mkdir -p ~deploy/.ssh
chown deploy:deploy ~deploy/.ssh
chmod 700 ~deploy/.ssh

Allow the ssh key you set for the droplet root user to log into the deploy user account:

cp ~root/.ssh/authorized_keys ~deploy/.ssh
chown deploy:deploy ~deploy/.ssh/authorized_keys
chmod 600 ~deploy/.ssh/authorized_keys

Exit the ssh session and connect again using the deploy user.

ssh -A deploy@web-server

If it doesn't work, check that the ssh key on your dev machine (.ssh/id_rsa.pub) is in the ~/.ssh/authorized_keys file for the deploy user and check the file permissions. Try with -vv or look at /var/log/auth.log on the server.

The -A flag on the ssh command gives the session on the server access to your local ssh keys. If your local user can access a GitHub repo, then you can do it on the server without having to set up keys on the server.

Make sure that the deploy user can run commands with sudo:

sudo -s
exit

Check out the app source

As the deploy user on the build machine, create the build dir:

mkdir -p ~/build

Check out the app source:

cd ~/build
git clone https://github.com/cogini/mix-deploy-example # or your app repo
cd mix-deploy-example

Install build dependencies

We generally use ASDF to manage build tools rather than OS packages. That allows us to precisely specify versions and install multiple versions at once. However, with simple applications it's fine to install Erlang, Elixir and Node.js from OS packages:

LANG=en_US.UTF-8 sudo bin/build-install-deps-ubuntu

See the instructions on the Elixir website for more details on installing Elixir and dependencies.

Create the database

Most apps use a database. You could install the database on the same Droplet as you run your app. It works fine and is cheaper, but then you have to manage the db. This guide uses Digital Ocean's managed databases service.

In the Digital Ocean UI, select Create → Databases.

  • Choose a database engine: Select PostgreSQL 11
  • Choose a cluster configuration: $15/month is fine
  • Choose a datacenter: Use the same data center as your droplet
  • Choose a unique database cluster name: The default name is fine

While the database is being created, in the “Getting started” section of the page, click the bullet point that says “Secure this database cluster.” Under “Restrict inbound connections” select your droplet and click “Allow these inbound sources only.” This ensures that only your application server can connect to the database.

Create app databases and users

If you are creating only one database per app, you can use the defaultdb database and doadmin user that the setup wizard created for you. However, it is better to create a separate database and database user for each app environment.

Configuration

Configuration for a Phoenix application can be split into three parts:

  • Build time

These settings are the same for all servers, though they may differ between dev, test and production. We handle this with the normal Phoenix config files in config/config.exs, config/test.exs and config/prod.exs.

Ideally we would be able to run the same software build in our staging and production environments, allowing us to test exactly the same thing as we will run. Settings in these config files are compiled into the release files, so for security they should not include secrets like db passwords.

  • Environment / per machine / secrets

These settings depend on the environment the application is running in, e.g. the hostname of the db server and secrets like the db password. We store these external to the application release and load them from files or a configuration system like AWS Systems Manager Parameter Store or etcd.

Distillery introduced the concept of config providers, which are modules that fetch your configuration from all sources and merge them into an Erlang term file called sys.config, which is then read by the BEAM. With Distillery, you could use the Mix.Config provider that would load an Elixir source file formatted for Mix.Config. This config provider module is not compatible with Mix releases. However, you can achieve the same result by putting your configuration code in the file config/releases.exs, which is evaluated on startup. To load an external Elixir configuration file, you can invoke the macro Config.import_config/1. Another good option is a TOML config file.

In this approach, config files are stored in the application's configuration directory under /etc, allowing it to be managed by startup scripts like mix_deploy, deploy-sync-config-s3, or configuration management tools like Ansible.

  • Runtime

These settings are dynamic and may change every time the application starts. For example, if we are running in an AWS auto scaling group, the IP address of the server normally changes every time it starts. We load them at runtime when the app starts using scripts like deploy-runtime-environment-file which reads it from cloud-init.

Building

On your build machine, build the app by running the build script:

bin/build

In addition to the normal Phoenix build steps, this command sets up the deploy scripts by running the following mix_systemd and mix_deploy commands:

# NOTE: you don't have to run these commands manually
mix systemd.init
mix systemd.generate

mix deploy.init
mix deploy.generate

The configuration is minimal. We just change the name of the OS user that the app runs under to app in config/prod.exs:

config :mix_deploy,
  app_user: "app",
  app_group: "app"

# Minimal
config :mix_systemd,
  app_user: "app",
  app_group: "app"

Prepare runtime configuration

Next we have to prepare a runtime config file with any variables that should be kept secret, such as database connection settings, API tokens, or your secret_key_base. These settings should not be included in the release. On your build machine, copy config/prod.secret.exs.sample somewhere outside the build directory and edit it to match the settings for your database.

cp config/prod.secret.exs.sample ~/prod.secret.exs
# Before Elixir 1.9, you would have to `use Mix.Config`
# That module has been deprecated in favor of Config
import Config

config :mix_deploy_example, MixDeployExample.Repo,
  username: "doadmin",
  password: "CHANGEME",
  database: "defaultdb",
  hostname: "db-postgresql-sfo2-xxx-do-user-yyy-0.db.ondigitalocean.com",
  port: 25060,
  ssl: true,
  pool_size: 15

config :mix_deploy_example, MixDeployExampleWeb.Endpoint,
  secret_key_base: "CHANGEME2"

You can generate a unique value for secret_key_base using this command:

mix phx.gen.secret

Build

Build the app and make a release:

bin/build

Initialize local system

Run this once to set up the system for the app, creating users and directories for releases, runtime configuration, etc.:

sudo bin/deploy-init-local

As this script changes group membership, you should log out and in again to reload user privileges.

Deploy the config

Copy secrets to the app runtime configuration directory:

cp ~/prod.secret.exs /etc/mix-deploy-example/config.exs
chown deploy:app /etc/mix-deploy-example/config.exs
chmod 644 /etc/mix-deploy-example/config.exs

Deploy the app

Deploy the release to the local machine:

# Extract release to target directory, creating current symlink
bin/deploy-release

# Run database migrations
bin/deploy-migrate

# Restart the systemd unit
sudo bin/deploy-restart

Check that it works

Make a request to the server:

curl -v http://localhost:4000/

You can get a console on the running release:

sudo -i -u app /srv/mix-deploy-example/bin/deploy-remote-console

You can also have a look at the logs:

sudo systemctl status mix-deploy-example
sudo journalctl -r -u mix-deploy-example

You can roll back the release with the following:

bin/deploy-rollback
sudo bin/deploy-restart

Configure the server to listen on port 80

Listening on port 4000 might be fine if it's behind a load balancer, otherwise we need to make the app available on port 80. There are two ways to do this:

After you complete this step, you should be able to access your website in the browser by navigating to your droplet's public IP address.

SSL

The steps necessary to get SSL set up with your Phoenix application depend on the approach that you took in the previous step. If you are forwarding ports using iptables, then you should set up SSL in your application's endpoint, as described in Phoenix docs. You can get an SSL certificate for free from Let's Encrypt.

If you are running behind an Nginx reverse proxy, you should instead set up SSL in Nginx. The necessary steps are described in Digital Ocean's tutorials.

How to prepare your Phoenix app for deployment

Following are the steps used to set up this repo. You can do the same to add it to your own project. This repo is built as a series of git commits, so you can see how it works step by step.

Generate Phoenix project

mix phx.new your_app
mix deps.get
cd assets && npm install && node node_modules/webpack/bin/webpack.js --mode development
  • Add mix.lock to git
  • Add package-lock.json to git

Configure releases

Elixir 1.9 has built in support for creating releases. For earlier versions, use the Distillery library.

Generate initial config files in the rel dir:

mix release.init

Check the rel directory into git.

In mix.exs, tell Mix not to include Windows executables in releases. In the main project configuration, add the option :releases:

def project do
  [
    app: :mix_deploy_example,
    version: "0.1.0",
    elixir: "~> 1.9",
    elixirc_paths: elixirc_paths(Mix.env()),
    compilers: [:phoenix, :gettext] ++ Mix.compilers(),
    start_permanent: Mix.env() == :prod,
    aliases: aliases(),
    deps: deps(),

    # add this line:
    releases: releases()
  ]
end

Then, in the same file, add a private function that returns your project's release configuration:

defp releases do
  [
    # change this to your application name
    mix_deploy_example: [
      include_executables_for: [:unix]
    ]
  ]
end

Tune the OS for performance

For optimal performance, it is recommended that you increase the default limits for open TCP ports and file handles. Follow the instructions in the post Tuning TCP ports for your Elixir app.

Add runtime config files

Loading runtime configuration from Elixir source files in Elixir 1.9 is very straightforward. The file config/releases.exs is copied into your release and evaluated at startup, and you can load your Elixir configuration file from within that file. Edit config/releases.exs:

import Config
import_config "/etc/mix-deploy-example/config.exs"

Create a config/prod.secret.exs.sample that you can use to generate production configuration files on the build server:

import Config

# Change these identifiers to ones specific to your application
config :mix_deploy_example, MixDeployExample.Repo,
  username: "CHANGEME",
  password: "CHANGEME",
  database: "CHANGEME",
  hostname: "CHANGEME",
  port: 5432,
  ssl: true,
  pool_size: 15

config :mix_deploy_example, MixDeployExampleWeb.Endpoint,
  secret_key_base: "CHANGEME2"

Add migrator module (optional)

In order for the bin/deploy-migrate script to work properly, you need to add a migrator module to your project. The instructions on how to do so are described in the post Running Ecto migrations in a release.

If your build server and production server are the same machine, you can also skip this step and just run your migrations with MIX_ENV=prod mix ecto.migrate.

Set up ASDF

Create a .tool-versions file in the root of your project, describing the versions of OTP, Elixir, and Node that you will be building with:

erlang 21.3
elixir 1.9.0
nodejs 10.16.0

Install mix_deploy and mix_systemd

Add libraries to deps from Hex:

{:mix_systemd, "~> 0.5.0"},
{:mix_deploy, "~> 0.5.0"}

Or from GitHub:

{:mix_systemd, github: "cogini/mix_systemd", override: true},
{:mix_deploy, github: "cogini/mix_deploy"},
end

Add rel/templates and bin/deploy-* to .gitignore:

echo '/rel/templates' >> .gitignore
echo '/bin/deploy-*' >> .gitignore

Copy build and utility scripts into your repo

Copy shell scripts from the bin/ directory of the mix-deploy-example repo to the bin/ directory of your project.

These scripts build your release or install the required dependencies:

  • build
  • build-install-asdf
  • build-install-asdf-deps-centos
  • build-install-asdf-deps-ubuntu
  • build-install-asdf-init
  • build-install-asdf-macos
  • build-install-deps-centos
  • build-install-deps-ubuntu

This script verifies that your application is running correctly:

  • validate-service

Check these scripts into git.

Configure for running in a release

In config/prod.exs, uncomment or add this line so that Phoenix can run correctly in a release:

config :phoenix, :serve_endpoints, true

In the same file, configure mix_deploy and mix_systemd to run your application as the app user. This step is mandatory:

config :mix_deploy,
  app_user: "app",
  app_group: "app"

# Minimal
config :mix_systemd,
  app_user: "app",
  app_group: "app"

Still in prod.exs, configure your application's endpoint to fetch port number from environment variables. The corresponding variable will be set by systemd:

config :your_app_name, YourAppNameWeb.Endpoint,
  http: [:inet6, port: System.get_env("PORT") || 4000],
  # ...

Confirm that everything compiles by building the app:

mix deps.get
mix deps.compile
mix compile

You should be able to run the app locally with:

# Create development database
mix ecto.create

# Compile assets with production settings
(cd assets && npm install && npm run deploy)

mix phx.server
open http://localhost:4000/

If everything seems to work, you can proceed with deployment just like you did with the mix-deploy-example sample application.