"Tower" by " Yijun Chen" on Flickr

Building a Gitlab and Ansible Tower (AWX) Demo in Vagrant with Ansible

TL;DR – I created a repository on GitHub‌ containing a Vagrantfile and an Ansible Playbook to build a VM running Docker. That VM hosts AWX (Ansible Tower’s upstream open-source project) and Gitlab.

A couple of years ago, a colleague created (and I enhanced) a Vagrant and Ansible playbook called “Project X” which would run an AWX instance in a Virtual Machine. It’s a bit heavy, and did a lot of things to do with persistence that I really didn’t need, so I parked my changes and kept an eye on his playbook…

Fast-forward to a week-or-so ago. I needed to explain what a Git/Ansible Workflow would look like, and so I went back to look at ProjectX. Oh my, it looks very complex and consumed a lot of roles that, historically, I’ve not been that impressed with… I just needed the basics to run AWX. Oh, and I also needed a Gitlab environment.

I knew that Gitlab had a docker-based install, and so does AWX, so I trundled off to find some install guides. These are listed in the playbook I eventually created (hence not listing them here). Not all the choices I made were inspired by those guides – I wanted to make quite a bit of this stuff “build itself”… this meant I wanted users, groups and projects to be created in Gitlab, and users, projects, organisations, inventories and credentials to be created in AWX.

I knew that you can create Docker Containers in Ansible, so after I’d got my pre-requisites built (full upgrade, docker installed, pip libraries installed), I add the gitlab-ce:latest docker image, and expose some ports. Even now, I’m not getting the SSH port mapped that I was expecting, but … it’s no disaster.

I did notice that the Gitlab service takes ages to start once the container is marked as running, so I did some more digging, and found that the uri module can be used to poll a URL. It wasn’t documented well how you can make it keep polling until you get the response you want, so … I added a PR on the Ansible project’s github repo for that one (and I also wrote a blog post about that earlier too).

Once I had a working Gitlab service, I needed to customize it. There are a bunch of Gitlab modules in Ansible but since a few releases back of Gitlab, these don’t work any more, so I had to find a different way. That different way was to run an internal command called “gitlab-rails”. It’s not perfect (so it doesn’t create repos in your projects) but it’s pretty good at giving you just enough to build your demo environment. So that’s getting Gitlab up…

Now I need to build AWX. There’s lots of build guides for this, but actually I had most luck using the README in their repository (I know, who’d have thought it!??!) There are some “Secrets” that should be changed in production that I’m changing in my script, but on the whole, it’s pretty much a vanilla install.

Unlike the Gitlab modules, the Ansible Tower modules all work, so I use these to create the users, credentials and so-on. Like the gitlab-rails commands, however, the documentation for using the tower modules is pretty ropey, and I still don’t have things like “getting your users to have access to your organisation” working from the get-go, but for the bulk of the administration, it does “just work”.

Like all my playbooks, I use group_vars to define the stuff I don’t want to keep repeating. In this demo, I’ve set all the passwords to “Passw0rd”, and I’ve created 3 users in both AWX and Gitlab – csa, ops and release – indicative of the sorts of people this demo I ran was aimed at – Architects, Operations and Release Managers.

Maybe, one day, I’ll even be able to release the presentation that went with the demo ;)

On a more productive note, if you’re doing things with the tower_ modules and want to tell me what I need to fix up, or if you’re doing awesome things with the gitlab-rails tool, please visit the repo with this automation code in, and take a look at some of my “todo” items! Thanks!!

Featured image is “Tower” by “Yijun Chen” on Flickr and is released under a CC-BY-SA license.

A web browser with the example.com web page loaded

Working around the fact that Ansible’s URI module doesn’t honour the no_proxy variable…

An Ansible project I’ve been working on has tripped me up this week. I’m working with some HTTP APIs and I need to check early whether I can reach the host. To do this, I used a simple Ansible Core Module which lets you call an HTTP URI.

- uri:
    follow_redirects: none
    validate_certs: False
    timeout: 5
    url: "http{% if ansible_https | default(True) %}s{% endif %}://{{ ansible_host }}/login"
  register: uri_data
  failed_when: False
  changed_when: False

This all seems pretty simple. One of the environments I’m working in uses the following values in their environment:

http_proxy="http://192.0.2.1:8080"
https_proxy="http://192.0.2.1:8080"
no_proxy="10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24"

And this breaks the uri module, because it tries to punt everything through the proxy if the “no_proxy” contains CIDR values (like 192.0.2.0/24) (there’s a bug raised for this)… So here’s my fix!

- set_fact:
    no_proxy_match: |
      {
        {% for no_proxy in (lookup('env', 'no_proxy') | replace(',', '') ).split() %}
          {% if no_proxy| ipaddr | type_debug != 'NoneType' %}
            {% if ansible_host | ipaddr(no_proxy) | type_debug != 'NoneType' %}
              "match": "True"
            {% endif %}
          {% endif %}
        {% endfor %}
      }

- uri:
    follow_redirects: none
    validate_certs: False
    timeout: 5
    url: "http{% if ansible_https | default(True) %}s{% endif %}://{{ ansible_host }}/login"
  register: uri_data
  failed_when: False
  changed_when: False
  environment: "{ {% if no_proxy_match.match | default(False) %}'no_proxy': '{{ ansible_host }}'{% endif %} }"

So, let’s break this down.

The key part to this script is that we need to override the no_proxy environment variable with the IP address that we’re trying to address (so that we’re not putting 16M addresses for 10.0.0.0/8 into no_proxy, for example). To do that, we use the exact same URI block, except for the environment line at the end.

In turn, the set_fact block steps through the no_proxy values, looking for IP Addresses to check ({% if no_proxy | ipaddr ... %}‌ says “if the no_proxy value is an IP Address, return it, but if it isn’t, return a ‘None’ value”) and if it’s an IP address or subnet mask, it checks to see whether the IP address of the host you’re trying to reach falls inside that IP Address or Subnet Mask ({% if ansible_host | ipaddr(no_proxy) ... %} says “if the ansible_host address falls inside the no_proxy range, then return it, otherwise return a ‘None’ value”). Both of these checks say “If this previous check returns anything other than a ‘None’ value, do the next thing”, and on the last check, the “next” thing is to set the flag ‘match’ to ‘true’. When we get to the environment variable, we say “if match is not true, it’s false, so don’t put a value in there”.

So that’s that! Yes, I could merge the set_fact block into the environment variable, but I do end up using that a fair amount. And really, if it was merged, that would be even MORE complicated to pick through.

I have raised a pull request on the Ansible project to update the documentation, so we’ll see whether we end up with people over here looking for ways around this issue. If so, let me know in the comments below! Thanks!!

Submitting tracks and shows to CCHits.net

I’ve been on a few podcasts to talk about CCHits.net, and people always ask how to submit tracks to CCHits.net. It used to be pretty complicated. Well, ok, not *that* complicated, but enough that it put some people off.

One of the major things I wanted to do with the re-write was to make it so anyone can submit a track to CCHits.net. Of course, it still has this reputation for being a bit difficult to get tracks in there, so I thought I’d document just how simple it actually is!

So, first things first. Go to http://cchits.net/admin and register yourself. CCHits.net uses OpenID for authentication, and I’ve picked three of the major OpenID providers as simple one-click buttons to register and later login with. Basically, OpenID asks the referring site to confirm you are a user on their system, and to return a unique value for you as a user on their service. CCHits.net requests only two details from that provider – your e-mail address (if you have it stored with that provider, and you permit the site to have it – although not all providers will provide that restriction), and a “secret key” which only CCHits.net and that provider know, so the next time you login with OpenID to CCHits.net – it’ll know it’s you coming back.

So, click on a button to login, assuming you have an account on one of the three main services, but if not, you may find this page from OpenID.net to be useful! To ensure the quality of tracks submitted is relatively good, CCHits.net verifies each submitter. This is done by sending an e-mail to show@cchits.net, quoting your UserID (as shown below)

I’ll then probably ask you a few questions, and mark your account to get access to the admin areas. Let’s assume that I’ve let you in!

For the purposes of this post, I’m submitting tracks to the site based on plays from a fantastic new show “Listener Feedback”. These tracks were played on a recent show where they reviewed the album “Doctor X” by “Fresh Body Shop”. They played 3 tracks, and here I’ll walk you through submitting the first of these tracks, and what that gives you.

After you log back in, you’ll get to the “Admin” page. From here you’ll initially want to use the “Retrieve or Upload a track” button, and complete the Track URL, but if you’ve been supplied an MP3, OGG (OGG Vorbis aka OGG Audio or OGA) or MP4 Audio track, you can browse to that file and submit it too. Bear in mind that CCHits.net makes every track it has available to it, available to anyone to download, so if you’ve been given a pre-release track, or the file should only be downloaded from the artist’s website – it shouldn’t be submitted to CCHits.net. We do encourage visitors to download from the originating site, but in cases where the site is not available, or the user just can’t be bothered, they can get the file from the site.

In the above box, you’d put the URL to get to the track and hit “Retrieve”. Just under the “Retrieve” button is a list of sites that CCHits.net currently understands how to retrieve data from – if the url isn’t on that list, you’ll have to upload the file yourself. The tracks at ListenerFeedback.net, while they refer to the MySpace URL for the artist, are actually available from Jamendo.com – so let’s find the first track from the show. The below screen shot shows the artist, album and tracks – the first track is called “Can’t get enough”

Let’s look for it on Jamendo.

Found it! We need the URL from the top there. Copy & paste it into CCHits.net, then hit Retrieve.

In this case, Fresh Body Shop have been played before, so CCHits.net compares a few different things, and asks whether this is the same artist that has been played before. Click “Select this artist” to put the artist into the site.

It’s a bit hard to make out all the details in the below screenshot, but notice there are a few interesting things.

  • Track and Artist names and URLs have three columns to use – “Set Default”, “Add new value” and “Delete value”. To enable track and artist matching, you can specify several alternative names and URLs for artists and tracks – for example, if the artist’s name is “An Artist” and the track is listed as “An Artist’s Debut Track – A Track!”, you should probably add the track name “A Track!”, and make that the default. Don’t remove it, otherwise the next time the track is added (if it is!), it might not be picked up by the de-duplicator. Likewise, if you’ve found a few different places to download a track from, add them all to the Track URLs, but pick one URL as the default – preferably one which the band tends to favour, or which would encourage the fans to donate money to them.
  • There’s a “Track Name Sounds” and “Artist Name Sounds” – these are because the system uses a text-to-speech engine to render the daily, weekly and monthly shows. It can’t just assume that the pronunciation is as-per the title or name, especially if there are abbreviations like “feat” or if this is a remix or version – for example “A Track (2012 Remix)” should be listed as “The 2012 remix of A Track”, or “A Track (feat. Joe Bloggs)” should be listed as “A Track, featuring Joe Bloggs”.
  • Lastly, there’s a “Not Safe For Work” flag. I am a father of a young child, and I want to be sure that any track I play for him is either work/family safe, or I at least know it’s a little raw before he hears it. This means that I’d like you to flag anything which contains swearing, makes reference to drugs or firearms, or strongly suggests sexual activity (including moans and grunts!). This doesn’t mean the tracks won’t be played – far from it, but it does mean that before these tracks are played, a little notice is played to say that it may not be considered work or family safe – it just gives people a chance to skip on for now!

Next to each of these fields is a “Go” button – click on that for each edit you make. If you’re just submitting tracks to CCHits.net – your work here is done! Excellent work, thanks for your help. You can see towards the end of this post what these tracks look like when they’re done. If you’re a podcaster or radio presenter, and want to show you played this track on one of your shows, you should click the button next to “Associate this track with a show”.

We’ll get some details to populate the show details on the site. Here is the Listener Feedback page we want to link to. As a minimum we need the URL for the show, but usually there is also a show title, so let’s capture that as well.

Let’s transfer those details into the “Add a track to a show” page. As this is the first track we’re putting in the show, we need to create the show. Put the details in, and click Go. Later tracks will list the show name and URL as non-editable fields, plus the “Go” button at the end of the line.

And that’s it, you’re done with that track. As I mentioned, this episode featured three tracks, so after submitting all of those three, after each track, you get the next screen shot, listing the tracks in the order you’ve played them (you’ll see why in a bit) including buttons to move them up, down and remove them from the show, an edit button for the track, and a vote URL. The Vote URL can be included in your show notes, if you want to point people back to CCHits.net – obviously, you may not be able to, or you may choose to keep your traffic on your own site – it’s entirely up to you – there are no requirements on you to tell people about CCHits, but you’ll get more from the site if you do (after all, we track vote clicks, so you can see whether your tracks are popular with your listeners from the show page).

Should you list the vote URL, and someone clicks on that link? Here’s what they’ll see.

And if they click “I like it”? Where to find out more about the artist (their default URL) and the track (the default URL), and the show notes for your show. They see the number of votes it’s received and the license it was released under, other shows it has been on, and most importantly for tracks from a show, the tracks which come before and after this one.

If you send someone to just the show page, you can see the show name, the details about the track and an “I like this track” button to vote on it. There’s also a QR code to take the listener to more details about the track but without making them vote on it. It also allows them to send their friends to the track if they like it. I realise this is one aspect of the site which needs some work, so if you’re a web developer and can help out, please get in touch!

If you visit the track, you’ll see the sharing QR code, the links to the track and artist, a vote button, details about the votes this track has received, plus details about the adjustments which are made (you’ll have to click through to the FAQ to find out more!). There’s information about the chart position, about where to download the file from, the license, and where else the track has been played on, and the split of votes for that track on those shows. Early votes on the site were not properly tracked, which is why this track in particular has lots of votes for “Non-show votes”, when they were probably daily show votes!

As an submitter on the same page, you’ll see a couple of other links. Notice the track URL is exactly the same! You get to edit the track with the track editor you saw above, and you get to add that track to your own show. If you inadvertently add a track that has already been played before, the site tries several different ways to catch this, and let you use the previously uploaded track in it’s place, via this very link here!

If you want to see all your shows, from the admin page, click on the “Show a list of the shows I created” button

To see your shows!

I hope you’ve found this useful, and consider submitting tracks to CCHits.net