When deploying applications, we we usually have the same basic architecture in different environments (dev, test, prod), but settings differ. Some settings are common to all the machines in the environment, e.g. the db server connection string. We need to vary the size of instances depending on the environment, and we need to keep application secrets like passwords by environment.

What we would like is to put a machine in multiple groups, setting some defaults for the whole system, then overriding them by role and environment. Unfortunately that doesn't work in Ansible. There are priority rules between var sources, but all groups are the same. The Ansible "best practice" (limitation) is that a variable should be defined in one and only one place.

If your environments are relatively static, e.g. dedicated servers, then you can do it as follows:

Use the all group to set defaults for all servers. Next set group-specific variables by server "role", which take priority over all. Finally, set host-specific vars, which take priority over group vars.

inventory/group_vars/all
inventory/group_vars/web-servers
inventory/group_vars/app-servers
inventory/host_vars/server-01

If you are using the Ansible vault (recommended), these are directories, so you end up with something like this:

inventory/group_vars/web-servers/vars.yml
inventory/group_vars/web-servers/vault.yml

Define your groups in inventory/hosts and add servers to them.

[web-servers]
server-01

[app-servers]
server-02

If you have environment-specific vars, then you can make:

inventory/group_vars/web-servers
inventory/group_vars/app-servers
inventory/group_vars/web-servers-prod
inventory/group_vars/app-servers-prod
inventory/group_vars/web-servers-stage
inventory/group_vars/app-servers-stage


[web-servers]
server-01
server-03

[app-servers]
server-02
server-04

[web-servers-prod]
server-01

[web-servers-stage]
server-03

[app-servers-prod]
server-02

[web-servers-stage]
server-04

You end up duplicating some variables to deal with the lack of hierarchy. You can also use AWS dynamic inventory to assign servers to roles using tags. It supports having lots of servers.

This system breaks down if you have lots of applications and environments, though, e.g. multiple copies of the same app in production for different customers. One of our customers has a dozen apps deployed to AWS, each running in dev/test/staging/demo/prod environments. In addition, they run production environments in multiple regions, (US, EU, China, etc.).

In this case, we use a different structure to keep the combinational explosion of variables under control.

Make a playbook that sets up the machine or app, e.g. playbooks/myapp/web-server.yml:

- name: Configure web server
  remote_user: ubuntu
  hosts: '*'
  become: true
  gather_facts: true
  vars_files:
    - vars/myapp/{{ env }}/app.yml
    - vars/myapp/{{ env }}/common.yml
    - vars/myapp/{{ env }}/datadog.yml
    - vars/myapp/{{ env }}/keys.yml
  roles:
    - {role: ubuntu-common}
    - role: datadog.datadog
      when: '"env == prod" or "env == demo"'

Then make vars files for each combination of app and env, e.g. vars/myapp/prod/app.yml.

The vars directory is relative to the playbook. So you it could be playbooks/vars/myapp/dev/app.yml or at the top level. In that case the vars_files would be ../vars/myapp/{{ env }}/app.yml.

Finally, call the playbook specifying the environment:

ansible-playbook -i "myapp-$ENV" --extra-vars "env=$ENV" playbooks/myapp/web-server.yml

Here we set an OS environment var ENV=prod which pulls in a separate inventory (which could also use a dynamic inventory script) and set the Ansible env var which loads right var files.

If you need to set up an instance per customer, you can have vars/myapp/a/app.yml and vars/myapp/b/app.yml. And you can share common vars like var/myapp/common/app.yml.

In all of these, the playbooks don't use a lot of conditional vars, though they can if necessary distinguish between e.g. dev and prod. Generally it's best to use roles with default vars set in e.g. roles/ubuntu-common/defaults/main.yml. The tasks can use something like when: '"env == prod"' or you can conditionally include a role as shown above.