Provisioning AWS EC2 Instances with Ansible and Automating Apache deployment with or without using Ansible Dynamic Inventory from Ubuntu 20.04 LTS
This article is being included in my book Provisioning to Amazon AWS using boto3 SDK for Python 3.
Pre-requisites
I’ll use Ubuntu 20.04 LTS.
Python 2 is required for Ansible.
Python 3 is required for our programs.
sudo apt install python2 python3 python3-pip # Install boto for Python 2 for Ansible (alternative way if pip install boto doesn't work for you) python2 -m pip install boto # Install Ansible sudo apt install ansible
If you want to use Dynamic Inventory
So you can use the Python 2 ec2.py and ec2.ini files, adding them as to the /etc/ansible with mask +x, to use the Dynamic Inventory.
You will need to have your credentials set.
I use Environment variables:
#!/bin/bash export ANSIBLE_HOST_KEY_CHECKING=false export AWS_ACCESS_KEY=AKIXXXXXXXXXXXXXOS export AWS_SECRET_KEY=e4dXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXY6F
Then use the calls inside the shell script, or assuming that the previous file was named credentiasl.sh use source credentials.sh
ec2.py is written in Python 2, so probably will fail for you as it is invoked by python and your default interpreter will be Python 3.
So edit the first line of /etc/ansible/ec2.py and add:
#!/bin/env python2
Once credentials.sh is sourced, then you can just invoke ec2.py to get the list of your Instances in a JSON format dumped by ec2.py
/etc/ansible/ec2.py --list
You can get that JSON file and load it and get the information you need, filtering by group.
You can call:
/etc/ansible/ec2.py --list > instances.json
Or you can run a Python program that escapes to shell and executes ec2.py –list and loads the Output as a JSON file.
I use my carleslibs here to escape to shell using my class SubProcessUtils. You can install them, they are Open Source, or you can code manually if you prefer importing subprocess Python library and catching the stdout, stderr.
import json from carleslibs import SubProcessUtils if __name__ == "__main__": s_command = "/etc/ansible/ec2.py" o_subprocess = SubProcessUtils() i_error_code, s_output, s_error = o_subprocess.execute_command_for_output(s_command, b_shell=True, b_convert_to_ascii=True, b_convert_to_utf8=False) if i_error_code != 0: print("Error escaping to shell!", i_error_code) print(s_error) exit(1) json = json.loads(s_output) d_hosts = json["_meta"]["hostvars"] for s_host in d_hosts: # You'll get a ip/hostnamename in s_host which is the key # You have to check for groups and the value for the key Name, in order to get the Name of the group # As an exercise, print(d_hosts[s_host]) and look for: # @TODO: Capture the s_group_name # @TODO: Capture the s_addres if s_group_name == "yourgroup": # This filters only the instances with your group name, as you want to create an inventory file just for them # That's because you don't want to launch the playbook for all the instances, but for those in your group name in the inventory file. a_hostnames.append(s_address) # After this you can parse you list a_hostnames and generate an inventory file yourinventoryfile # The [ec2hosts] in your inventory file must match the hosts section in your yaml files # You'll execute your playbook with: # ansible-playbook -i yourinventoryfile youryamlfile.yaml
So an example of a yaml to install Apache2 in Ubuntu 20.04 LTS spawned instances , let’s call it install_apache2.yaml would be:
--- - name: Update web servers hosts: ec2hosts remote_user: ubuntu tasks: - name: Ensure Apache is at the latest version apt: name: apache2 state: latest update_cache: yes become: yes
As you can see the section hosts: in the YAML playbook matches the [ec2hosts] in your inventory file.
You can choose to have your private key certificate .pem file in /etc/ansible/ansible.cfg or if you want to have different certificates per host, add them after the ip/address in your inventory file, like in this example:
[localhost] 127.0.0.1 [ec2hosts] 63.35.186.109 ansible_ssh_private_key_file=ansible.pem
The ansible.pem certificate must have restricted permissions, for example chmod 600 ansible.pem
Then you end by running:
ansible-playbook -i yourinventoryfile install_ubuntu.yaml
If you don’t want to use Dynamic Directory
The first method is to use add_host to print in the screen the properties form the ec2 Instances provisioned.
The trick is to escape to shell, executing ansible-playbook and capturing the output, then parsing the text looking for the ‘public_ip:’
This is the Python 3 code I created:
class AwesomeAnsible: def extract_public_ips_from_text(self, s_text=""): """ Extracts the addresses returned by Ansible :param s_text: :return: Boolean for success, Array with the Ip's """ b_found = False a_ips = [] i_count = 0 while True: i_count += 1 if i_count > 20: print("Breaking look") break s_substr = "'public_ip': '" i_first_pos = s_text.find(s_substr) if i_first_pos > -1: s_text_sub = s_text[i_first_pos + len(s_substr):] # Find the ending delimiter i_second_pos = s_text_sub.find("'") if i_second_pos > -1: b_found = True s_ip = s_text_sub[0:i_second_pos] a_ips.append(s_ip) s_text_sub = s_text_sub[i_second_pos:] s_text = s_text_sub continue # No more Ip's break return b_found, a_ips
Then you’ll use with something like:
# Catching the Ip's from the output b_success, a_ips = self.extract_public_ips_from_text(s_output) if b_success is True: print("Public Ips:") s_ips = "" for s_ip in a_ips: print(s_ip) s_ips = s_ips + self.get_ip_text_line_for_inventory(s_ip) print("Adding Ips to group1_inventory file") self.o_fileutils.append_to_file("group1_inventory", s_ips) print()
The get_ip_text_line_for_inventory_method() returns a line for the inventory file, with the ip and the key to use separated by a tab (\t):
def get_ip_text_line_for_inventory(self, s_ip, s_key_path="ansible.pem"): """ Returns the line to add to the inventory, with the Ip and the keypath """ return s_ip + "\tansible_ssh_private_key_file=" + s_key_path + "\n"
Once you have the inventory file, like this below, you can execute the playbook for your group of hosts:
[localhost] 127.0.0.1 [ec2hosts] 63.35.186.109 ansible_ssh_private_key_file=ansible.pem
ansible-playbook -i yourinventoryfile install_ubuntu.yaml
Alternative way parsing with awk and grep
You can run this Bash Shell Script to get only the public ips when you provision to Amazon AWS EC2 the Instances from your group named group1 in this case:
./launch_aws_instances-group1.sh | grep "public_ip" | awk '{ print $13; }' | tr -d "'," 52.213.232.199
In this example 52.213.232.199 is the Ip from the Instance I provisioned.
So to put it together, from a Python file I generate this Bash file and I escape to shell to execute it:
#!/bin/bash export ANSIBLE_HOST_KEY_CHECKING=false export AWS_ACCESS_KEY=AKIXXXXXXXXXXXXXOS export AWS_SECRET_KEY=e4dXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXY6F # Generate a new Inventory File echo "[localhost]" > group1_inventory echo "127.0.0.1" >> group1_inventory echo "[ec2hosts]" >> group1_inventory ansible-playbook -i group1_inventory launch_aws_instances-group1.yaml
I set again the credentials because as this Bash Shell Script is invoked from Python, there are not sourced.
The trick in here is that the launch_aws_instances-group1.yaml file has a task to add the hosts to Ansible’s in memory inventory, and to print the information.
That output is what I scrap from Python and then I use extract_public_ips_from_text() showed before.
So my launch_aws_instances-group1.yaml (which I generate from Python customizing the parameter) looks like this:
# launch_aws_instances.yaml - hosts: localhost connection: local gather_facts: False vars: s_keypair_name: "ansible" s_instance_type: "t1.micro" s_image: "ami-08edbb0e85d6a0a07" s_group: "ansible" s_region: "eu-west-1" i_exact_count: 1 s_tag_name: "ansible_group1" tasks: - name: Provision a set of instances ec2: key_name: "{{ s_keypair_name }}" region: "{{ s_region }}" group: "{{ s_group }}" instance_type: "{{ s_instance_type }}" image: "{{ s_image }}" wait: true exact_count: "{{ i_exact_count }}" count_tag: Name: "{{ s_tag_name }}" instance_tags: Name: "{{ s_tag_name }}" register: ec2_ips - name: Add all instance public IPs to host group add_host: hostname={{ item.public_ip }} groups=ec2hosts loop: "{{ ec2_ips.instances }}"
In this case I use t1.micro cause I provision to EC2-Classic and not to the default VPC, otherwise I would use t2.micro.
So I have a Security Group named ansible created in Amazon AWS EC2 console as EC2-Classic, and not as VPC.
In this Security group I opened the Inbound HTTP Port and the SSH port for the Ip from I’m provisioning, so Ansible can SSH using the Key ansible.pem
The Public Key has been created and named ansible as well (section key_name under ec2).
The Image used is Ubuntu 20.04 LTS (free tier) for the region eu-west-1 which is my wonderful Ireland.
For the variables (vars) I use the MT Notation, so the prefixes show exactly what we are expecting s_ for Strings i_ for Integers and I never have collisions with reserved names.
It is very important to use the count_tag and instance_tags with the name of the group, as the actions will be using that group name. Remember the idempotency.
The task with the add_host is the one that makes the information for the instances to be displayed, like in this screenshot.
Rules for writing a Comment