Building dynamic inventory with grouping on ansible using EC2 inventory plugin

Some background

So, going to the point, I spent way too much time understanding how to organize a full dynamic inventory and grouping that inventory. All the examples and docs just show how to configure and use directly that host via something like ansible-playbook -i "my-bullshit-example" playbook.yml, but that’s bullshit and not a usual real world example.

Since one of the projects I’m working on requires this feature (infrastructure is simple, but it get’s destroyed and re-created from time to time, and no, I won’t be using a fixed IP since this infra part is mostly for testing) I started searching for the latest method on getting a dynamic inventory.

I use ansible 2.10 and python 3.8 since I like keeping things updated on my personal projects, at work we’ve been using not the dynamic inventory plugin, BUT the script and this can get really confusing.

The old method are the scripts, newest is plugins, however there’s “backward” compatibility, but in reality it does not (At least on AWS ec2 script)

My inventory setup

I currently run 3 different environments, so basically the thing goes like this:

Current inventory setup

Notice the aws_ec2.yml? I’ll explain what is it on the next step. If you don’t understand how this is organized, refer to ansible inventory documentation for host and group organization and hosts.yml setup (although my hosts.yml is a bit different to the shown on the docs)

TO WAR (ノಠ益ಠ)ノ彡┻━┻

Install the plugin

Installing the official aws ec2 inventory plugin requires to run:

ansible-galaxy collection install amazon.aws

This will install the amazon collection (includes modules as well) on the home from the user you are running the command, after that you need to configure the file I mentioned before, aws_ec2.yml the naming scheme of the file should be like *aws_ec2.(yml|yaml), it’s a requirement from the plugin

Configure aws_ec2.yml

This is the configuration file and you should configure it to suit your needs, in my case I want to:

• run only eu-west-3
• configure the credentials on the file (DO NOT DO THIS IF THIS FILE IS GONNA BE EXPOSED SOMEWHERE, use env variables instead)
• group by tags
• filter by env staging
• Get ip, dns name, tag Name tag env and private IP

My file results in something like this:

Check this docs to see what every option does

Place this file inside your inventory, at the same level as the hosts.yml, you can have different aws_ec2.yml for every env, and for example filter by other AWS tag like production instead of staging.

Warning about ansible.cfg & enable_plugins

At some places I’ve seen people modifying ansible.cfg and adding the enable_plugins, DO NOT DO THAT, ansible will enable the plugin by default, and if you specify only that plugin, your inventory will not work and ansible will just say it cannot parse hosts.yml and it result in something weird like this:

inventory listing with enable_plugins set only for ec2

First run does not work (shows ec2 inventory, but is not grouped), when I run it at the root level. But it works when I run it inside the inventory folder… That is why, inside inventory the config file is the ansible default, but on the root folder I had the custom ansible.cfg

I spent 3 hours on this, so yeah…

Configuring hosts.yml

So, the file looks like this:

Basically is tag_Key_value… For example, if you had an aws instance with the tag Env and value prod, it would look like tag_Env_prod beware of using values like prod-test since they will be converted to prod_test.

Testing time

Okay, so congrats for getting here, to test if your config works, you should run

ansible-inventory -i inventory/your-env --graph

You should get a listing with grouped hosts based on what you had configured on those 2 files. You can run

ansible -m ping -i inventory/your-env host

To test if the host is reachable as well. Run with -vvvv to see some details as well.

That’s all folks!

References:

• EC2 plugin — https://docs.ansible.com/ansible/latest/collections/amazon/aws/aws_ec2_inventory.html
• Script inventory — https://docs.ansible.com/ansible/latest/user_guide/intro_dynamic_inventory.html
• Plugin inventory — https://docs.ansible.com/ansible/latest/plugins/inventory.html