Ansible Automation: A Comprehensive Guidebook Part 2

Published Sep 08, 2017Last updated Mar 06, 2018
Ansible Automation: A Comprehensive Guidebook Part 2

Table of Contents

Part 1 Continued

In the first part of this series, we looked at how easy it is to work with Ansible Playbooks, introduced the concept of idempotence, and showed you how to assign values to variables and how to use those variables in the context of a playbook. In this installment, we will look at more efficient ways of writing your playbooks.

It is very possible to write a playbook as a really long file with all of the logic you need for your deployment. In fact, this is probably how you will start using playbooks. There is nothing wrong with that.

As you become more proficient with Ansible, you will want to start to break things up and reuse your playbooks. Task includes are a good way to pull in plays from other files. Since handlers are tasks too, you could also include handlers from the handlers section just as well.

When you start to think of things this way, the concept of modeling systems begins to take form in your head. You will start to think about what your systems are and instead of trying to make this system be like that other system, you will begin to define your systems based on their purpose. You will apply your configurations to "these machines that are app servers" or "to those machines that are cache servers." This is called 'encapsulation' and it is very much the same concept that allows you and I to drive a car without knowing how it actually works. Very powerful stuff indeed.

Roles in Ansible encapsulate a great deal of functionality with the idea of task and play includes which are combined to form highly reusable abstractions. You can now focus on the big picture! But if you want to drill into all of the details you still can if and when you need to.

Intermediate Ansible

Task and Play Includes

Tasks and plays both use the include keyword but implement the functionality quite differently. Where you position and what an include contains will determine these differences. Generally speaking, you should keep the following rules in mind:

  1. If the include is inside a play iy can only be a task include and it can only include a list of tasks.
  2. If the include is at the top level, it can only include plays.
  3. A task include can appear anywhere a task can, but a play include cannot be inside other plays, only alongside them at the same level.
  4. While task includes can take other parameters and have the included tasks inherit them, play includes are very limited and most directives do not work.

This all sounds more complicated than what it really is. Let's look at some examples to illustrate these points.

A task include file simply contains a flat lists of tasks:

---
# File tasks/foo.yml

- name: placeholder foo
  command: /bin/foo

- name: placeholder bar
  command: /bin/bar

Include directives look something like this and can be mixed with regular tasks in a playbook:

 tasks:

  - include: tasks/foo.yml

Now, as I mentioned in Part 1 of this series, variables are simply referenced like so in Ansible:

{{ some_variable }}

Includes can also be used in the handlers section. You might want to make a handlers.yml file that looks like this:

---
# File handlers/handlers.yml
- name: restart apache
  service: name=apache state=restarted

And in the main playbook file, just include it at the bottom of a play.

handlers:
  - include: handlers/handlers.yml

Includes can be mixed in along with regular tasks and handlers.

Includes can also be used in plays to bring in other plays:

---
# File playbook.yml
- name: this is a play at the top level of a file
  hosts: all
  remote_user: root

  tasks:

  - name: say hi
    tags: foo
    shell: echo "hi..."

- include: load_balancers.yml
- include: webservers.yml
- include: dbservers.yml

As you can see, Ansible is quite powerful.

Now that I have told you about handlers and tasks and how you perform task includes and play includes you may be asking yourself, "How am I supposed to organize all of these things in a way that makes sense?" Easy. Let me tell you about Ansible Roles.

Ansible Roles

Ansible roles are nothing more than a set of variables, tasks, and handlers organized, or encapsulated in a directory structure that Ansible expects. Organizing your content in this way also enables you to reuse your roles as well as to share them with other users easily. Let's take a look at the structure of a playbook directory that incorporates roles.

site.yml
webservers.yml
fooservers.yml
roles/
   common/
     files/
     templates/
     tasks/
     handlers/
     vars/
     defaults/
     meta/
   webservers/
     files/
     templates/
     tasks/
     handlers/
     vars/
     defaults/
     meta/     

The Site.yml file would look like this:

---
- hosts: webservers
  roles:
     - common
     - webservers

By now, you should be starting to understand just how powerful the concept of roles can be. Now, why don't you try to picture having the freedom to pass parameters to your roles, or allowing your roles to execute based on a known fact about your systems. Facts are things like CPU type, amount of RAM, or OS version installed in your systems. Young grasshopper, I think a light just lit inside your head! And this is just the beginning!

OK, I know that your head must be spinning like a tornado. But please, stop that. We must continue our review of the basics. Now, let me tell you about a few other things that you must keep in mind while working with Ansible roles.

Default Variables

It is always a good idea to include default variables within your roles simply because you might decide to reuse a role and forget to define a required variable within your playbook. It happens. By including the minimum amount of variables within your roles, and setting those variables to sane, default values, you will allow your role to execute successfully every time.

The way to do this in Ansible is to simply define all of your variables in the defaults/main.yml file, just like you would any other variable.

Dependencies

But wait. "Ansible makes things easy!" you say? Well, yes, it most certainly does. And in case you are wondering, there is still room for improvement. Enter role dependencies.

In Ansible, role dependencies provide the mechanism to pull in other roles as part of your own by simply inserting a few lines in the meta/main.yml file like this:

# File meta/main.yml
---
dependencies:
  - { role: common, some_parameter: 3 }
  - { role: apache, apache_port: 80 }
  - { role: postgres, dbname: blarg, other_parameter: 12 }

As you can see, you can also parameterize role dependencies. And yes, you can also use variables as parameters that can be passed to your dependencies. Is that cool or what?

The Jinja2 Ninja

Jinja2 is the preferred templating language of Ansible’s template module. It is a very simple Python template language that is generally readable and easy to write.

If you have never used Jinja2, don't worry. And if you have used Jinja2 before, great! You can rest assured that your knowledge will easily transfer to the world of Ansible. Either way, it will be good for you to know that Ansible greatly expands the number of filters available in its implementation of Jinja2. There is also one plugin that is not part of vanilla Jinja2: lookups

In other words, Ansible Jinja2 templates may not be backward-compatible with vanilla Jinja2. You can generally bring vanilla Jinja2 templates into Ansible without a problem.

Also keep in mind that all templating happens on the Ansible controller before the task is sent and executed on the target machine. This is done to minimize the requirements on the target machine (Ansible only requires jinja2 on the controller machine) and also to minimize the amount of information needed to pass to the target machine in order for it to complete the task. Said a different way, we just don't give the target machine a copy of all the data the controller has access to.

We will not get too deep into Jinja2 but certainly encourage you to seek more information online as Jinja2 is at the core of Ansible.

Jinja2 Filters

In Jinja2, filters are used for transforming data inside a template expression and many filters are shipped with Jinja2.

{{ some_variable | to_nice_json }}
{{ some_variable | to_nice_yaml }}

The default behavior of Ansible is to fail when a variable it needs is not defined. You can change this behavior by forcing variables to be defined before the playbook is executed.

{{ variable | mandatory }}

Or, you can also assign a default value to a variable.

{ some_variable | default(5) }}

List filters are available to allow you to do many things.

# Gets the minimum value from a list of numbers.
{{ list1 | min }}

# Gets the maxinum value from a list of numbers.
{{ [3, 4, 2] | max }}

# Gets a unique set from a list.
{{ list1 | unique }}

# Gets a union of two lists
{{ list1 | union(list2) }}

# Gets the difference of 2 lists (items in 1 that don’t exist in 2).
{{ list1 | difference(list2) }}

There is also a random number filter that is similar to the one included in vanilla Jinja2 but can also generate a random number based on. a range.

# Gets random number from list.
"{{ ['a','b','c']|random }}"

# Gets random number from 1 to 100 in steps of 10
{{ 100 |random(step=10) }}

There is a filter that will shuuffle lists and return a random number from it.

# Gets a random number from an existing list.
{{ ['a','b','c']|shuffle }}

It is also possible to shuffle a list idempotent if we have a seed.

{{ ['a','b','c']|shuffle(seed=inventory_hostname) }}

And we can also do math with Jinja2 filters.

# Gets the logarithm.
{{ myvar | log }}

# Gets the base 10 logarithm
{{ myvar | log(10) }}

# Gets the power of 2.
{{ myvar | pow(2) }}

# Square root
{{ myvar | root }}

Jinja2 can also filter IP addresses.

# Tests if a string is a valid IP address.
{{ myvar | ipaddr }}

# Require a specific IP protocol version.
{{ myvar | ipv4 }}

# Extract the IP address value
{{ '192.0.2.1/24' | ipaddr('address') }}

We can also do hashing with Jinja2.

# Gets the sha1 hash of a string.
{{ 'test1'|hash('sha1') }}

# Gets an md5 hash of a string.
{{ 'test1'|hash('md5') }}

# Get a string checksum.
{{ 'test2'|checksum }}

Jinja2 Tests

What if you wanted to evaluate whether or not an expression was true or false? You might be able to do that with a very creative use of Jinja2 filters but you don't have to do that. Instead, you should use Jinja2 Tests. Remember, just like Jinja2 Filters execute on the controller machine, so do Jinja2 Tests.

The "match" or "search" filter matches strings against a substring or a regex

vars:
  url: "http://example.com/users/foo/resources/bar"

tasks:
    - debug: "msg='matched pattern 1'"
      when: url | match("http://example.com/users/.*/resources/.*")

    - debug: "msg='matched pattern 2'"
      when: url | search("/users/.*/resources/.*")

    - debug: "msg='matched pattern 3'"
      when: url | search("/users/")

To compare a version number, such as checking if the ansible_distribution_version version is greater than or equal to ‘12.04’, you can use the version_compare filter.

{{ sample_version_var | version_compare('1.0', operator='lt', strict=True) }}

To see if a list includes or is included by another list, you can use ‘issubset’ and ‘issuperset’.

vars:
    a: [1,2,3,4,5]
    b: [2,3]
tasks:
    - debug: msg="A includes B"
      when: a|issuperset(b)

    - debug: msg="B is included in A"
      when: b|issubset(a)

To see if a list includes or is included by another list, you can use ‘issubset’ and ‘issuperset’.

vars:
    a: [1,2,3,4,5]
    b: [2,3]
tasks:
    - debug: msg="A includes B"
      when: a|issuperset(b)

    - debug: msg="B is included in A"
      when: b|issubset(a)

The following tasks are illustrative of the tests meant to check the status of tasks.

tasks:

  - shell: /usr/bin/foo
    register: result
    ignore_errors: True

  - debug: msg="it failed"
    when: result|failed

In most cases you'll want a handler, but if you want to do something right now, this is nice.

  - debug: msg="it changed"
    when: result|changed

  - debug: msg="it succeeded in Ansible >= 2.1"
    when: result|succeeded

  - debug: msg="it succeeded"
    when: result|success

  - debug: msg="it was skipped"
    when: result|skipped

Jinja2 Lookups

The developers of Ansible devised a method for allowing you to access information from outside sources via Jinja2 plugins. These lookup plugins also execute on the controller machine and can be used to access all kinds of information, from reading the filesystem, to contacting AWS, GCloud, Azure, or just about any other service you have access to. Many useful lookup plugins are included with Ansible right out-of-the-box and if you have a need for something custom-tailored to your needs, you can always roll up your own plugin.

Reading the filesystem

- hosts: all
  vars:
     contents: "{{ lookup('file', '/etc/foo.txt') }}"

  tasks:

     - debug: msg="the value of foo.txt is {{ contents }}"

Reading Passwords

- hosts: all

  tasks:

    - name: create a mysql user with a random password
      mysql_user:
        name: "{{ client }}"
        password: "{{ lookup('password', 'credentials/' + client + '/' + tier + '/' + role + '/mysqlpassword length=15') }}"
        priv: "{{ client }}_{{ tier }}_{{ role }}.*:ALL"

    # (...)

The example below shows the contents of a CSV file named elements.csv with information about the periodic table of elements:

Symbol,Atomic Number,Atomic Mass
H,1,1.008
He,2,4.0026
Li,3,6.94
Be,4,9.012
B,5,10.81

We can use the csvfile plugin to look up the atomic number or atomic of Lithium by its symbol:

- debug: msg="The atomic number of Lithium is {{ lookup('csvfile', 'Li file=elements.csv delimiter=,') }}"
- debug: msg="The atomic mass of Lithium is {{ lookup('csvfile', 'Li file=elements.csv delimiter=, col=2') }}"

Jinja2Fu

Ansible also includes lookup plugins to read INI files, to do Credstash lookups, Passwordstore lookups, DNS lookups, MongoDB lookups, and you can even use these plugins to iterate through sets of data. If you have not yet realized this, Ansible's Jinja2 templating engine is powerful and extremely expressive and lies at the core of Ansible. It's power cannot be understated. Proficiency in Jinja2 is a must if you are thinking about seriously learning to work with Ansible. However, I have found that just a basic understanding of Jinja2 is enough. Regardless, I strongly encourage you to seek the Ansible and Jinja2 Documentation online and learn as much as you can about it.

Ansible Conditionals

Some times, the steps you will want Ansible to take will depend on the value of a variable, or the value of a fact, or the outcome of the previous task. Don't forget, you can also group your systems in unlimited ways an match against that specific criteria as well. We can use any and all of these things (and many more) to control the execution flow of our playbooks. Let's review some of the available conditionals we have in Ansible.

WHEN

There will be times when you will want to skip a step on a particular host. Perhaps the steps in your playbook are specific to Debian-like distributions. Or maybe your role only applies to Windows systems. Whatever the case may be, the when clause makes it a snap to do so.

tasks:
  - name: "shut down Debian flavored systems"
    command: /sbin/shutdown -t now
    when: ansible_os_family == "Debian"
    # note that Ansible facts and vars like ansible_os_family can be used
    # directly in conditionals without double curly braces

The when clause, like anything else in. Ansible, is quite flexible and extremely powerful. It can be combined with with_items to create conditional loops.

tasks:
    - command: echo {{ item }}
      with_items: [ 0, 2, 4, 6, 8, 10 ]
      when: item > 5

If you need to skip the whole task depending on the loop variable being defined, use the |default filter to provide an empty iterator:

- command: echo {{ item }}
  with_items: "{{ mylist|default([]) }}"
  when: item > 5

If using with_dict which does not take a list:

- command: echo {{ item.key }}
  with_dict: "{{ mydict|default({}) }}"
  when: item.value > 5

Custom Fact Loading

It is very easy to provide our own set of facts about a system or group(s) of systems. Simply call your custom module that loads all of your custom facts and those variables will then be avaialble for any other tasks that follow.

tasks:
    - name: gather site specific fact data
      action: site_facts
    - command: /usr/bin/thingy
      when: my_custom_fact_just_retrieved_from_the_remote_system == '1234'

When for Roles and Includes

If you have a bunch of tasks that all have the same conditional statement in common then you can attach the confitional to a task include as illustrated below. All the tasks will be evaluated and the conditional is applied to each and every task.

- include: tasks/sometasks.yml
  when: "'reticulating splines' in output"

Or you can use this same technique with a role.

- hosts: webservers
  roles:
     - { role: debian_stock_config, when: ansible_os_family == 'Debian' }

Here are a few links for those of you who just can't wait for the next installment!

Part 3 Coming Soon

My goal is to encourage you to take Ansible for a spin and see what it can do for you. If I have succeeded in sparking that curiousity in just one of you, I will have succeeded. So, please do let me know if I have motivated you to try Ansible and please make sure to tell me how you are using Ansible in your own environment. I would really like to know.

Oh, and by the way, the last part in the series will be out before the end of October, 2017. I promise!


I can be reached at my Gmail account.!

Discover and read more posts from Jorge Vazquez
get started