"Key mess" by "Alper Çuğun" on Flickr

Making Ansible Keys more useful in complex and nested data structures

One of the things I miss about Jekyll when I’m working with Ansible is the ability to fragment my data across multiple files, but still have it as a structured *whole* at the end.

For example, given the following directory structure in Jekyll:

+ _data
+---+ members
|   +--- member1.yml
|   +--- member2.yml
+---+ groups
    +--- group1.yml
    +--- group2.yml

The content of member1.yml and member2.yml will be rendered into site.data.members.member1 and site.data.members.member2 and likewise, group1 and group2 are loaded into their respective variables.

This kind of structure isn’t possible in Ansible, because all the data files are compressed into one vars value that we can read. To work around this on a few different projects I’ve worked on, I’ve ended up doing the following:

- set_fact:
    my_members: |-
        {%- for var in vars | dict2items -%}
          {%- if var.key | regex_search(my_regex) is not none -%}
            "{{ var.key | regex_replace(my_regex, '') }}": 
              {%- if var.value | string %}"{% endif -%}
              {{ var.value }}
              {%- if var.value | string %}"{% endif %},
          {%- endif -%}
        {%- endfor -%}
    my_regex: '^member_'

So, what this does is to step over all the variables defined (for example, in host_vars\*, group_vars\*, from the gathered facts and from the role you’re in – following Ansible’s loading precedence), and then checks to see whether the key of that variable name (e.g. “member_i_am_a_member” or “member_1”) matches the regular expression (click here for more examples). If it does, the key (minus the regular expression matching piece [using regex_replace]) is added to a dictionary, and the value attached. If the value is actually a string, then it wraps it in quotes.

So, while this doesn’t give me my expressive data structure that Jekyll does (no site.data.members.member1.somevalue for me), I do at least get to have my_members.member1.somevalue if I put the right headers in! :)

I’ll leave extending this model for doing other sorts of building variables out (for example, something like if var.value['variable_place'] | default('') == 'my_members.member' + current_position) to the reader to work out how they could use something like this in their workflows!

Featured image is “Key mess” by “Alper Çuğun” on Flickr and is released under a CC-BY license.

"Seca" by "Olearys" on Flickr

Getting Started with Terraform on Azure

I’m strongly in the “Ansible is my tool, what needs fixing” camp, when it comes to Infrastructure as Code (IaC) but, I know there are other tools out there which are equally as good. I’ve been strongly advised to take a look at Terraform from HashiCorp. I’m most familiar at the moment with Azure, so this is going to be based around resources available on Azure.

Late edit: I want to credit my colleague, Pete, for his help getting started with this. While many of the code samples have been changed from what he provided me with, if it hadn’t been for these code samples in the first place, I’d never have got started!

Late edit 2: This post was initially based on Terraform 0.11, and I was prompted by another colleague, Jon, that the available documentation still follows the 0.11 layout. 0.12 was released in May, and changes how variables are reused in the code. This post now *should* follow the 0.12 conventions, but if you spot something where it doesn’t, check out this post from the Terraform team.

As with most things, there’s a learning curve, and I struggled to find a “simple” getting started guide for Terraform. I’m sure this is a failing on my part, but I thought it wouldn’t hurt to put something out there for others to pick up and see if it helps someone else (and, if that “someone else” is you, please let me know in the comments!)


You need an Azure account for this. This part is very far outside my spectrum of influence, but I’m assuming you’ve got one. If not, look at something like Digital Ocean, AWS or VMWare :) For my “controller”, I’m using Windows Subsystem for Linux (WSL), and wrote the following notes about getting my pre-requisites.

Building the file structure

One quirk with Terraform, versus other tools like Ansible, is that when you run one of the terraform commands (like terraform init, terraform plan or terraform apply), it reads the entire content of any file suffixed “tf” in that directory, so if you don’t want a file to be loaded, you need to either move it out of the directory, comment it out, or rename it so it doesn’t end .tf. By convention, you normally have three “standard” files in a terraform directory – main.tf, variables.tf and output.tf, but logically speaking, you could have everything in a single file, or each instruction in it’s own file. Because this is a relatively simple script, I’ll use this standard layout.

The actions I’ll be performing are the “standard” steps you’d perform in Azure to build a single Infrastructure as a Service (IAAS) server service:

  • Create your Resource Group (RG)
  • Create a Virtual Network (VNET)
  • Create a Subnet
  • Create a Security Group (SG) and rules
  • Create a Public IP address (PubIP) with a DNS name associated to that IP.
  • Create a Network Interface (NIC)
  • Create a Virtual Machine (VM), supplying a username and password, the size of disks and VM instance, and any post-provisioning instructions (yep, I’m using Ansible for that :) ).

I’m using Visual Studio Code, but almost any IDE will have integrations for Terraform. The main thing I’m using it for is auto-completion of resource, data and output types, also the fact that control+clicking resource types opens your browser to the documentation page on terraform.io.

So, creating my main.tf, I start by telling it that I’m working with the Terraform AzureRM Provider (the bit of code that can talk Azure API).

This simple statement is enough to get Terraform to load the AzureRM, but it still doesn’t tell Terraform how to get access to the Azure account. Use az login from a WSL shell session to authenticate.

Next, we create our basic resource, vnet and subnet resources.

But wait, I hear you cry, what are those var.something bits in there? I mentioned before that in the “standard” set of files is a “variables.tf” file. In here, you specify values for later consumption. I have recorded variables for the resource group name and location, as well as the VNet name and subnet name. Let’s add those into variables.tf.

When you’ve specified a resource, you can capture any of the results from that resource to use later – either in the main.tf or in the output.tf files. By creating the resource group (called “rg” here, but you can call it anything from “demo” to “myfirstresourcegroup”), we can consume the name or location with azurerm_resource_group.rg.name and azurerm_resource_group.rg.location, and so on. In the above code, we use the VNet name in the subnet, and so on.

After the subnet is created, we can start adding the VM specific parts – a security group (with rules), a public IP (with DNS name) and a network interface. I’ll create the VM itself later. So, let’s do this.

BUT WAIT, what’s that ${trimspace(data.http.icanhazip.body)}/32 bit there?? Any resources we want to load from the terraform state, but that we’ve not directly defined ourselves needs to come from somewhere. These items are classed as “data” – that is, we want to know what their values are, but we aren’t *changing* the service to get it. You can also use this to import other resource items, perhaps a virtual network that is created by another team, or perhaps your account doesn’t have the rights to create a resource group. I’ll include a commented out data block in the overall main.tf file for review that specifies a VNet if you want to see how that works.

In this case, I want to put the public IP address I’m coming from into the NSG Rule, so I can get access to the VM, without opening it up to *everyone*. I’m not that sure that my IP address won’t change between one run and the next, so I’m using the icanhazip.com service to determine my IP address. But I’ve not defined how to get that resource yet. Let’s add it to the main.tf for now.

So, we’re now ready to create our virtual machine. It’s quite a long block, but I’ll pull certain elements apart once I’ve pasted this block in.

So, this is broken into four main pieces.

  • Virtual Machine Details. This part is relatively sensible. Name RG, location, NIC, Size and what happens to the disks when the machine powers on. OK.
name                             = "iaas-vm"
location                         = azurerm_resource_group.rg.location
resource_group_name              = azurerm_resource_group.rg.name
network_interface_ids            = [azurerm_network_interface.iaasnic.id]
vm_size                          = "Standard_DS1_v2"
delete_os_disk_on_termination    = true
delete_data_disks_on_termination = true
  • Disk details.
storage_image_reference {
  publisher = "Canonical"
  offer     = "UbuntuServer"
  sku       = "18.04-LTS"
  version   = "latest"
storage_os_disk {
  name              = "iaas-os-disk"
  caching           = "ReadWrite"
  create_option     = "FromImage"
  managed_disk_type = "Standard_LRS"
  • OS basics: VM Hostname, username of the first user, and it’s password. Note, if you want to use an SSH key, this must be stored for Terraform to use without passphrase. If you mention an SSH key here, as well as a password, this can cause all sorts of connection issues, so pick one or the other.
os_profile {
  computer_name  = "iaas"
  admin_username = var.ssh_user
  admin_password = var.ssh_password
os_profile_linux_config {
  disable_password_authentication = false
  • And lastly, provisioning. I want to use Ansible for my provisioning. In this example, I have a basic playbook stored locally on my Terraform host, which I transfer to the VM, install Ansible via pip, and then execute ansible-playbook against the file I uploaded. This could just as easily be a git repo to clone or a shell script to copy in, but this is a “simple” example.
provisioner "remote-exec" {
  inline = ["mkdir /tmp/ansible"]

  connection {
    type     = "ssh"
    host     = azurerm_public_ip.iaaspubip.fqdn
    user     = var.ssh_user
    password = var.ssh_password

provisioner "file" {
  source = "ansible/"
  destination = "/tmp/ansible"

  connection {
    type     = "ssh"
    host     = azurerm_public_ip.iaaspubip.fqdn
    user     = var.ssh_user
    password = var.ssh_password

provisioner "remote-exec" {
  inline = [
    "sudo apt update > /tmp/apt_update || cat /tmp/apt_update",
    "sudo apt install -y python3-pip > /tmp/apt_install_python3_pip || cat /tmp/apt_install_python3_pip",
    "sudo -H pip3 install ansible > /tmp/pip_install_ansible || cat /tmp/pip_install_ansible",
    "ansible-playbook /tmp/ansible/main.yml"

  connection {
    type     = "ssh"
    host     = azurerm_public_ip.iaaspubip.fqdn
    user     = var.ssh_user
    password = var.ssh_password

This part of code is done in three parts – create upload path, copy the files in, and then execute it. If you don’t create the upload path, it’ll upload just the first file it comes to into the path specified.

Each remote-exec and file provisioner statement must include the hostname, username and either the password, or SSH private key. In this example, I provide just the password.

So, having created all this lot, you need to execute the terraform workload. Initially you do terraform init. This downloads all the provisioners and puts them into the same tree as these .tf files are stored in. It also resets the state of the terraform discovered or created datastore.

Next, you do terraform plan -out tfout. Technically, the tfout part can be any filename, but having something like tfout marks it as clearly part of Terraform. This creates the tfout file with the current state, and whatever needs to change in the Terraform state file on it’s next run. Typically, if you don’t use a tfout file within about 20 minutes, it’s probably worth removing it.

Finally, once you’ve run your plan stage, now you need to apply it. In this case you execute terraform apply tfout. This tfout is the same filename you specified in terraform plan. If you don’t include -out tfout on your plan (or even run a plan!) and tfout in your apply, then you can skip the terraform plan stage entirely.

When I ran this, with a handful of changes to the variable files, I got this result:

Once you’re done with your environment, use terraform destroy to shut it all down… and enjoy :)

The full source is available in the associated Gist. Pull requests and constructive criticism are very welcome!

Featured image is “Seca” by “Olearys” on Flickr and is released under a CC-BY license.

"Tower" by " Yijun Chen" on Flickr

Building a Gitlab and Ansible Tower (AWX) Demo in Vagrant with Ansible

TL;DR – I created a repository on GitHub‌ containing a Vagrantfile and an Ansible Playbook to build a VM running Docker. That VM hosts AWX (Ansible Tower’s upstream open-source project) and Gitlab.

A couple of years ago, a colleague created (and I enhanced) a Vagrant and Ansible playbook called “Project X” which would run an AWX instance in a Virtual Machine. It’s a bit heavy, and did a lot of things to do with persistence that I really didn’t need, so I parked my changes and kept an eye on his playbook…

Fast-forward to a week-or-so ago. I needed to explain what a Git/Ansible Workflow would look like, and so I went back to look at ProjectX. Oh my, it looks very complex and consumed a lot of roles that, historically, I’ve not been that impressed with… I just needed the basics to run AWX. Oh, and I also needed a Gitlab environment.

I knew that Gitlab had a docker-based install, and so does AWX, so I trundled off to find some install guides. These are listed in the playbook I eventually created (hence not listing them here). Not all the choices I made were inspired by those guides – I wanted to make quite a bit of this stuff “build itself”… this meant I wanted users, groups and projects to be created in Gitlab, and users, projects, organisations, inventories and credentials to be created in AWX.

I knew that you can create Docker Containers in Ansible, so after I’d got my pre-requisites built (full upgrade, docker installed, pip libraries installed), I add the gitlab-ce:latest docker image, and expose some ports. Even now, I’m not getting the SSH port mapped that I was expecting, but … it’s no disaster.

I did notice that the Gitlab service takes ages to start once the container is marked as running, so I did some more digging, and found that the uri module can be used to poll a URL. It wasn’t documented well how you can make it keep polling until you get the response you want, so … I added a PR on the Ansible project’s github repo for that one (and I also wrote a blog post about that earlier too).

Once I had a working Gitlab service, I needed to customize it. There are a bunch of Gitlab modules in Ansible but since a few releases back of Gitlab, these don’t work any more, so I had to find a different way. That different way was to run an internal command called “gitlab-rails”. It’s not perfect (so it doesn’t create repos in your projects) but it’s pretty good at giving you just enough to build your demo environment. So that’s getting Gitlab up…

Now I need to build AWX. There’s lots of build guides for this, but actually I had most luck using the README in their repository (I know, who’d have thought it!??!) There are some “Secrets” that should be changed in production that I’m changing in my script, but on the whole, it’s pretty much a vanilla install.

Unlike the Gitlab modules, the Ansible Tower modules all work, so I use these to create the users, credentials and so-on. Like the gitlab-rails commands, however, the documentation for using the tower modules is pretty ropey, and I still don’t have things like “getting your users to have access to your organisation” working from the get-go, but for the bulk of the administration, it does “just work”.

Like all my playbooks, I use group_vars to define the stuff I don’t want to keep repeating. In this demo, I’ve set all the passwords to “Passw0rd”, and I’ve created 3 users in both AWX and Gitlab – csa, ops and release – indicative of the sorts of people this demo I ran was aimed at – Architects, Operations and Release Managers.

Maybe, one day, I’ll even be able to release the presentation that went with the demo ;)

On a more productive note, if you’re doing things with the tower_ modules and want to tell me what I need to fix up, or if you’re doing awesome things with the gitlab-rails tool, please visit the repo with this automation code in, and take a look at some of my “todo” items! Thanks!!

Featured image is “Tower” by “Yijun Chen” on Flickr and is released under a CC-BY-SA license.

A web browser with the example.com web page loaded

Working around the fact that Ansible’s URI module doesn’t honour the no_proxy variable…

An Ansible project I’ve been working on has tripped me up this week. I’m working with some HTTP APIs and I need to check early whether I can reach the host. To do this, I used a simple Ansible Core Module which lets you call an HTTP URI.

- uri:
    follow_redirects: none
    validate_certs: False
    timeout: 5
    url: "http{% if ansible_https | default(True) %}s{% endif %}://{{ ansible_host }}/login"
  register: uri_data
  failed_when: False
  changed_when: False

This all seems pretty simple. One of the environments I’m working in uses the following values in their environment:


And this breaks the uri module, because it tries to punt everything through the proxy if the “no_proxy” contains CIDR values (like (there’s a bug raised for this)… So here’s my fix!

- set_fact:
    no_proxy_match: |
        {% for no_proxy in (lookup('env', 'no_proxy') | replace(',', '') ).split() %}
          {% if no_proxy| ipaddr | type_debug != 'NoneType' %}
            {% if ansible_host | ipaddr(no_proxy) | type_debug != 'NoneType' %}
              "match": "True"
            {% endif %}
          {% endif %}
        {% endfor %}

- uri:
    follow_redirects: none
    validate_certs: False
    timeout: 5
    url: "http{% if ansible_https | default(True) %}s{% endif %}://{{ ansible_host }}/login"
  register: uri_data
  failed_when: False
  changed_when: False
  environment: "{ {% if no_proxy_match.match | default(False) %}'no_proxy': '{{ ansible_host }}'{% endif %} }"

So, let’s break this down.

The key part to this script is that we need to override the no_proxy environment variable with the IP address that we’re trying to address (so that we’re not putting 16M addresses for into no_proxy, for example). To do that, we use the exact same URI block, except for the environment line at the end.

In turn, the set_fact block steps through the no_proxy values, looking for IP Addresses to check ({% if no_proxy | ipaddr ... %}‌ says “if the no_proxy value is an IP Address, return it, but if it isn’t, return a ‘None’ value”) and if it’s an IP address or subnet mask, it checks to see whether the IP address of the host you’re trying to reach falls inside that IP Address or Subnet Mask ({% if ansible_host | ipaddr(no_proxy) ... %} says “if the ansible_host address falls inside the no_proxy range, then return it, otherwise return a ‘None’ value”). Both of these checks say “If this previous check returns anything other than a ‘None’ value, do the next thing”, and on the last check, the “next” thing is to set the flag ‘match’ to ‘true’. When we get to the environment variable, we say “if match is not true, it’s false, so don’t put a value in there”.

So that’s that! Yes, I could merge the set_fact block into the environment variable, but I do end up using that a fair amount. And really, if it was merged, that would be even MORE complicated to pick through.

I have raised a pull request on the Ansible project to update the documentation, so we’ll see whether we end up with people over here looking for ways around this issue. If so, let me know in the comments below! Thanks!!

Submitting tracks and shows to CCHits.net

I’ve been on a few podcasts to talk about CCHits.net, and people always ask how to submit tracks to CCHits.net. It used to be pretty complicated. Well, ok, not *that* complicated, but enough that it put some people off.

One of the major things I wanted to do with the re-write was to make it so anyone can submit a track to CCHits.net. Of course, it still has this reputation for being a bit difficult to get tracks in there, so I thought I’d document just how simple it actually is!

So, first things first. Go to http://cchits.net/admin and register yourself. CCHits.net uses OpenID for authentication, and I’ve picked three of the major OpenID providers as simple one-click buttons to register and later login with. Basically, OpenID asks the referring site to confirm you are a user on their system, and to return a unique value for you as a user on their service. CCHits.net requests only two details from that provider – your e-mail address (if you have it stored with that provider, and you permit the site to have it – although not all providers will provide that restriction), and a “secret key” which only CCHits.net and that provider know, so the next time you login with OpenID to CCHits.net – it’ll know it’s you coming back.

So, click on a button to login, assuming you have an account on one of the three main services, but if not, you may find this page from OpenID.net to be useful! To ensure the quality of tracks submitted is relatively good, CCHits.net verifies each submitter. This is done by sending an e-mail to show@cchits.net, quoting your UserID (as shown below)

I’ll then probably ask you a few questions, and mark your account to get access to the admin areas. Let’s assume that I’ve let you in!

For the purposes of this post, I’m submitting tracks to the site based on plays from a fantastic new show “Listener Feedback”. These tracks were played on a recent show where they reviewed the album “Doctor X” by “Fresh Body Shop”. They played 3 tracks, and here I’ll walk you through submitting the first of these tracks, and what that gives you.

After you log back in, you’ll get to the “Admin” page. From here you’ll initially want to use the “Retrieve or Upload a track” button, and complete the Track URL, but if you’ve been supplied an MP3, OGG (OGG Vorbis aka OGG Audio or OGA) or MP4 Audio track, you can browse to that file and submit it too. Bear in mind that CCHits.net makes every track it has available to it, available to anyone to download, so if you’ve been given a pre-release track, or the file should only be downloaded from the artist’s website – it shouldn’t be submitted to CCHits.net. We do encourage visitors to download from the originating site, but in cases where the site is not available, or the user just can’t be bothered, they can get the file from the site.

In the above box, you’d put the URL to get to the track and hit “Retrieve”. Just under the “Retrieve” button is a list of sites that CCHits.net currently understands how to retrieve data from – if the url isn’t on that list, you’ll have to upload the file yourself. The tracks at ListenerFeedback.net, while they refer to the MySpace URL for the artist, are actually available from Jamendo.com – so let’s find the first track from the show. The below screen shot shows the artist, album and tracks – the first track is called “Can’t get enough”

Let’s look for it on Jamendo.

Found it! We need the URL from the top there. Copy & paste it into CCHits.net, then hit Retrieve.

In this case, Fresh Body Shop have been played before, so CCHits.net compares a few different things, and asks whether this is the same artist that has been played before. Click “Select this artist” to put the artist into the site.

It’s a bit hard to make out all the details in the below screenshot, but notice there are a few interesting things.

  • Track and Artist names and URLs have three columns to use – “Set Default”, “Add new value” and “Delete value”. To enable track and artist matching, you can specify several alternative names and URLs for artists and tracks – for example, if the artist’s name is “An Artist” and the track is listed as “An Artist’s Debut Track – A Track!”, you should probably add the track name “A Track!”, and make that the default. Don’t remove it, otherwise the next time the track is added (if it is!), it might not be picked up by the de-duplicator. Likewise, if you’ve found a few different places to download a track from, add them all to the Track URLs, but pick one URL as the default – preferably one which the band tends to favour, or which would encourage the fans to donate money to them.
  • There’s a “Track Name Sounds” and “Artist Name Sounds” – these are because the system uses a text-to-speech engine to render the daily, weekly and monthly shows. It can’t just assume that the pronunciation is as-per the title or name, especially if there are abbreviations like “feat” or if this is a remix or version – for example “A Track (2012 Remix)” should be listed as “The 2012 remix of A Track”, or “A Track (feat. Joe Bloggs)” should be listed as “A Track, featuring Joe Bloggs”.
  • Lastly, there’s a “Not Safe For Work” flag. I am a father of a young child, and I want to be sure that any track I play for him is either work/family safe, or I at least know it’s a little raw before he hears it. This means that I’d like you to flag anything which contains swearing, makes reference to drugs or firearms, or strongly suggests sexual activity (including moans and grunts!). This doesn’t mean the tracks won’t be played – far from it, but it does mean that before these tracks are played, a little notice is played to say that it may not be considered work or family safe – it just gives people a chance to skip on for now!

Next to each of these fields is a “Go” button – click on that for each edit you make. If you’re just submitting tracks to CCHits.net – your work here is done! Excellent work, thanks for your help. You can see towards the end of this post what these tracks look like when they’re done. If you’re a podcaster or radio presenter, and want to show you played this track on one of your shows, you should click the button next to “Associate this track with a show”.

We’ll get some details to populate the show details on the site. Here is the Listener Feedback page we want to link to. As a minimum we need the URL for the show, but usually there is also a show title, so let’s capture that as well.

Let’s transfer those details into the “Add a track to a show” page. As this is the first track we’re putting in the show, we need to create the show. Put the details in, and click Go. Later tracks will list the show name and URL as non-editable fields, plus the “Go” button at the end of the line.

And that’s it, you’re done with that track. As I mentioned, this episode featured three tracks, so after submitting all of those three, after each track, you get the next screen shot, listing the tracks in the order you’ve played them (you’ll see why in a bit) including buttons to move them up, down and remove them from the show, an edit button for the track, and a vote URL. The Vote URL can be included in your show notes, if you want to point people back to CCHits.net – obviously, you may not be able to, or you may choose to keep your traffic on your own site – it’s entirely up to you – there are no requirements on you to tell people about CCHits, but you’ll get more from the site if you do (after all, we track vote clicks, so you can see whether your tracks are popular with your listeners from the show page).

Should you list the vote URL, and someone clicks on that link? Here’s what they’ll see.

And if they click “I like it”? Where to find out more about the artist (their default URL) and the track (the default URL), and the show notes for your show. They see the number of votes it’s received and the license it was released under, other shows it has been on, and most importantly for tracks from a show, the tracks which come before and after this one.

If you send someone to just the show page, you can see the show name, the details about the track and an “I like this track” button to vote on it. There’s also a QR code to take the listener to more details about the track but without making them vote on it. It also allows them to send their friends to the track if they like it. I realise this is one aspect of the site which needs some work, so if you’re a web developer and can help out, please get in touch!

If you visit the track, you’ll see the sharing QR code, the links to the track and artist, a vote button, details about the votes this track has received, plus details about the adjustments which are made (you’ll have to click through to the FAQ to find out more!). There’s information about the chart position, about where to download the file from, the license, and where else the track has been played on, and the split of votes for that track on those shows. Early votes on the site were not properly tracked, which is why this track in particular has lots of votes for “Non-show votes”, when they were probably daily show votes!

As an submitter on the same page, you’ll see a couple of other links. Notice the track URL is exactly the same! You get to edit the track with the track editor you saw above, and you get to add that track to your own show. If you inadvertently add a track that has already been played before, the site tries several different ways to catch this, and let you use the previously uploaded track in it’s place, via this very link here!

If you want to see all your shows, from the admin page, click on the “Show a list of the shows I created” button

To see your shows!

I hope you’ve found this useful, and consider submitting tracks to CCHits.net