Managing app secrets with Ansible

By Jake Morrison in DevOps on Sat 16 June 2018

In web applications we usually have a few things that are sensitive, e.g. the login to the production database or API keys used to access a third party API. We need to be particularly careful about how we manage these secrets, as they may allow attackers to access data without going through the application itself.

There are trade-offs in managing secrets, depending on the size of the organization.

For a small team of developers who are also the admins, then we implicitly trust our devs. We may store the secrets on our dev machine and push them from there to the app servers. It's better not to have secrets in the build environment, though, particularly if it's a third party CI service.

We need to keep the secrets separate from the build, loading them separately on the target system. That might mean putting them in a separate file or reading them at runtime from an external source like an S3 bucket or AWS Parameter Store. Access is controlled by IAM instance roles.

For secure applications like health care or finance, we need to tightly control access to production systems. We can restrict access to your ops team. Ideally nobody would log into production systems, and if they do, there is an audit log.

Ansible vault

The Ansible automation tool a vault function which we can use to store keys. It automates the process of encrypting variable data so we can check it into source control, and only people with the password can read it. It's great for simple deployments with small teams. It has very few moving parts and dependencies, while being reasonably secure.

The following shows describes how you can use the vault.

Configuring the vault key

First, generate a vault key and put it in the file vault.key:

openssl rand -hex 16

You can specify the password when you are running a playbook with the --vault-password-file vault.key option, or you can make the vault password always available by setting it in ansible.cfg:

vault_password_file = vault.key

Defining secrets

There are two ways to store secrets in Ansible variable files. Either we can encrypt the file as a whole, or we can embed encrypted data inline.

To create an encrypted variable file:

ansible-vault create --vault-id vault.key inventory/group_vars/web_servers/secrets.yml

Add variables normally. When you save the file, it will be encrypted. Later, edit the file like this:

ansible-vault edit --vault-id vault.key inventory/group_vars/web_servers/secrets.yml

To encrypt a single variable inline:

openssl rand -hex 32 | ansible-vault encrypt_string --vault-id vault.key --stdin-name 'db_pass'

That generates encrypted data like:

db_pass: !vault |

Copy that into a standard variable file.

Generating templates

Now, you can use Ansible's template function to create a config file template. The template file includes the variables, and Ansible will automatically decrypt vault variables and insert them into the template.

For example, here is a template which configures an Elixir Phoenix app:

[{{ elixir_app_name }}."{{ elixir_app_module }}Web.Endpoint"]
secret_key_base = "{{ secret_key_base }}"

[{{ elixir_app_name }}."{{ elixir_app_module }}.Repo"]
username = "{{ db_user }}"
password = "{{ db_pass }}"
database = "{{ db_name }}"
hostname = "{{ db_host }}"
ssl = {{ db_ssl }}
pool_size = {{ db_pool_size }}

The Ansible task could generate it on the production server:

- name: Create config.toml
  template: src=etc/app/config.toml.j2 dest=/etc/app/config.toml owner={{ app_user }} group={{ app_group }} mode=0644

Generating config files to S3

When deploying in the cloud, you can generate the app config file to an S3 bucket. When the app starts up, it can then sync the config file from the S3 bucket.

# Generate app config and upload to S3 bucket
# ansible-playbook -v -u $USER --extra-vars "env=$ENV" playbooks/$APP/config-app.yml -D

- name: Generate config file from template and upload to S3
  hosts: localhost
  gather_facts: no
  connection: local
    app_name: foo
    comp: app
    file_format: toml
    input_template: ../../templates/{{ app_name }}/{{ comp }}/config.{{ file_format }}.j2
    output_file: config.{{ file_format }}
    - ../../vars/{{ app_name }}/{{ env }}/common.yml
    - ../../vars/{{ app_name }}/{{ env }}/db-app.yml
    - ../../vars/{{ app_name }}/{{ env }}/app.yml
    - ../../vars/{{ app_name }}/{{ env }}/app-secrets.yml
    - name: Create tempfile
        state: file
      register: temp_file

    # - debug: var=temp_file.path

    - name: Fill template to tempfile
        src: "{{ input_template }}"
        dest: "{{ temp_file.path }}"
      no_log: true

    - name: Put config to S3
        bucket: "{{ config_bucket }}"
        object: "{{ config_bucket_prefix }}/{{ output_file }}"
        src: "{{ temp_file.path }}"
        mode: put

    - name: Delete tempfile
        state: absent
        path: "{{ temp_file.path }}"