"LEGO Factory Playset" from Brickset on Flickr

Building Azure Environments in Ansible

Recently, I’ve been migrating my POV (proof of value) and POC (proof of concept) environment from K5 to Azure to be able to test vendor products inside Azure. I ran a few tests to build the environment using the native tools (the powershell scripts) and found that the Powershell way of delivering Azure environments seems overly complicated… particularly as I’m comfortable with how Ansible works.

To be fair, I also need to look at Terraform, but that isn’t what I’m looking at today :)

So, let’s start with the scaffolding. Any Ansible Playbook which deals with creating virtual machines needs to have some extra modules installed. Make sure you’ve got ansible 2.7 or later and the python azure library 2.0.0 or later (you can get both with pip for python).

Next, let’s look at the group_vars for this playbook.

This file has several pieces. We define the project settings (anything prefixed project_ is a project setting), including the prefix used for all resources we create (in this case “env01“), and a standard password used for all VMs we create (in this case “My$uper$ecret$Passw0rd“).

Next we define the standard images to load from the Marketplace. You can extend this with other images, these are just the “easiest” ones that I’m most familiar with (your mileage may vary). Next up is the networks to build inside the VNet, and lastly we define the actual machines we want to build. If you’ve got questions about any of the values we define here, just let me know in the comments below :)

Next, we’ll start looking at the playbook (this has been exploded out – the full playbook is also in the gist).

Here we start by pulling in the variables we might want to override, and we do this by reading system environment variables (ANSIBLE_PREFIX and BREAKGLASS) and using them if they’re set. If they’re not, use the project defaults, and if that hasn’t been set, use some pre-defined values… and then tell us what they are when we’re running the tasks (those are the debug: lines).

This block is where we create our “Static Assets” – individual items that we will be consuming later. This shows a clear win here over the Powershell methods endorsed by Microsoft – here you can create a Resource Group (RG) as part of the playbook! We also create a single Storage Account for this RG and a single VNET too.

These creation rules are not suitable for production use, as this defines an “Any-Any” Security group! You should tailor your security groups for your need, not for blanket access in!

This is where things start to get a bit more interesting – We’re using the “async/async_status” pattern here (and the rest of these sections) to start creating the resources in parallel. As far as I can tell, sometimes you’ll get a case where the async doesn’t quite get set up fast enough, then the async_status can’t track the resources properly, but re-running the playbook should be enough to sort that out, without slowing things down too much.

But what are we actually doing with this block of code? A UDR is a “User Defined Route” or routing table for Azure. Effectively, you treat each network interface as being plumbed directly to the router (none of this “same subnet broadcast” stuff works here!) so you can do routing at the router for all the networks.

By default there are some existing network routes (stuff to the internet flows to the internet, RFC1918 addresses are dropped with the exception of any RFC1918 addresses you have covered in your VNETs, and each of your subnets can reach each other “directly”). Adding a UDR overrides this routing table. The UDRs we’re creating here are applied at a subnet level, but currently don’t override any of the existing routes (they’re blank). We’ll start putting routes in after we’ve added the UDRs to the subnets. Talking of which….

Again, this block is not really suitable for production use, and assumes the VNET supernet of /8 will be broken down into several /24’s. In the “real world” you might deliver a handful of /26’s in a /24 VNET… or you might even have lots of disparate /24’s in the VNET which are then allocated exactly as individual /24 subnets… this is not what this model delivers but you might wish to investigate further!

Now that we’ve created our subnets, we can start adding the routing table to the UDR. This is a basic one – add a 0.0.0.0/0 route (internet access) from the “protected” network via the firewall. You can get a lot more specific than this – most people are likely to want to add the VNET range (in this case 10.0.0.0/8) via the firewall as well, except for this subnet (because otherwise, for example, 10.0.0.100 trying to reach 10.0.0.101 will go via the firewall too).

Without going too much into the intricacies of network architecture, if you are routing your traffic between subnets to the firewall, it’s probably better to get an appliance with more interfaces, so you can route traffic across the appliance, rather than going across a single interface as this will halve your traffic bandwidth (it’s currently capped 1Gb/s – so 500Mb/s).

Having mentioned “The Internet” – let’s give our firewall a public IP address, and create the rest of the interfaces as well.

This script creates a public IP address by default for each interface unless you explicitly tell it not to (see lines 40, 53 and 62 in the group_vars file I rendered above). You could easily turn this around by changing the lines which contain this:

item.1.public is not defined or (item.1.public is defined and item.1.public == 'true')

into lines which contain this:

item.1.public is defined and item.1.public == 'true'

OK, having done all that, we’re now ready to build our virtual machines. I’ve introduced a “Priority system” here – VMs with priority 0 go first, then 1, and 2 go last. The code snippet below is just for priority 0, but you can easily see how you’d extrapolate that out (and in fact, the full code sample does just that).

There are a few blocks here to draw attention to :) I’ve re-jigged them a bit here so it’s clearer to understand, but when you see them in the main playbook they’re a bit more compact. Let’s start with looking at the Network Interfaces section!

network_interfaces: |
  [
    {%- for nw in item.value.ports -%}
      '{{ prefix }}{{ item.value.name }}port{{ nw.subnet.name }}'
      {%- if not loop.last -%}, {%- endif -%} 
    {%- endfor -%}
  ]

In this part, we loop over the ports defined for the virtual machine. This is because one device may have 1 interface, or four interfaces. YAML is parsed to make a JSON variable, so here we can create a JSON variable, that when the YAML is parsed it will just drop in. We’ve previously created all the interfaces to have names like this PREFIXhostnamePORTsubnetname (or aFW01portWAN in more conventional terms), so here we construct a JSON array, like this: ['aFW01portWAN'] but that could just as easily have been ['aFW01portWAN', 'aFW01portProtect', 'aFW01portMGMT', 'aFW01portSync']. This will then attach those interfaces to the virtual machine.

Next up, custom_data. This section is sometimes known externally as userdata or config_disk. My code has always referred to it as a “Provision Script” – hence the variable name in the code below!

custom_data: |
  {%- if item.value.provision_script is defined and item.value.provision_script != '' -%}
    {%- include(item.value.provision_script) -%}
  {%- elif item.value.image.provision_script is defined and item.value.image.provision_script != '' -%}
    {%- include(item.value.image.provision_script) -%}
  {%- else -%}
    {{ omit }}
  {%- endif -%}

Let’s pick this one apart too. If we’ve defined a provisioning script file for the VM, include it, if we’ve defined a provisioning script file for the image (or marketplace entry), then include that instead… otherwise, pretend that there’s no “custom_data” field before you submit this to Azure.

One last quirk to Azure, is that some images require a “plan” to go with it, and others don’t.

plan: |
  {%- if item.value.image.plan is not defined -%}{{ omit }}{%- else -%}
    {'name': '{{ item.value.image.sku }}',
     'publisher': '{{ item.value.image.publisher }}',
     'product': '{{ item.value.image.offer }}'
    }
  {%- endif -%}

So, here we say “if we’ve not got a plan, omit the value being passed to Azure, otherwise use these fields we previously specified. Weird huh?

The very last thing we do in the script is to re-render the standard password we’ve used for all these builds, so that we can check them out!

Want to review this all in one place?

Here’s the link to the full playbook, as well as the group variables (which should be in ./group_vars/all.yml) and two sample userdata files (which should be in ./userdata) for an Ubuntu machine (using cloud-init) and one for a FortiGate Firewall.

All the other files in that gist (prefixes from 10-16 and 00) are for this blog post only, and aren’t likely to work!

If you do end up using this, please drop me a note below, or star the gist! That’d be awesome!!

Image credit: “Lego Factory Playset” from Flickr by “Brickset” released under a CC-BY license. Used with Thanks!

Troubleshooting FortiGate API issues with the CLI?

One of my colleagues has asked me for some help with an Ansible script he’s writing to push some policy to a cloud hosted FortiGate appliance. Unfortunately, he kept getting some very weird error messages, like this one:

fatal: [localhost]: FAILED! => {"changed": false, "meta": {"build": 200, "error": -651, "http_method": "PUT", "http_status": 500, "mkey": "vip8080", "name": "vip", "path": "firewall", "revision": "36.0.0.10745196634707694665.1544442857", "serial": "CENSORED", "status": "error", "vdom": "root", "version": "v6.0.3"}, "msg": "Error in repo"}

This is using Fortinet’s own Ansible Modules, which, in turn use the fortiosapi python module.

This same colleague came across a post on the Fortinet Developer Network site (access to the site requires vendor approval), which said “this might be an internal bug, but to debug it, use the following”

fgt # diagnose debug enable

fgt # diagnose debug cli 8
Debug messages will be on for 30 minutes.

And then run your API commands. Your error message will be surfaced there… so here’s mine! (Mapped port doesn’t match extport in a vip).

0: config firewall vip
0: edit "vip8080"
0: unset src-filter
0: unset service
0: set extintf "port1"
0: set portforward enable
0: unset srcintf-filter
0: set mappedip "192.0.2.1-192.0.2.1"
0: unset extport
0: set extport 8080-8081
0: unset mappedport
0: set mappedport 8080
-651: end

Creating Self Signed certificates in Ansible

In my day job, I sometimes need to use a self-signed certificate when building a box. As I love using Ansible, I wanted to make the self-signed certificate piece something that was part of my Ansible workflow.

Here follows a bit of basic code that you could use to work through how the process of creating a self-signed certificate would work. I would strongly recommend using something more production-ready (e.g. LetsEncrypt) when you’re looking to move from “development” to “production” :)

---
- hosts: localhost
  vars:
  - dnsname: your.dns.name
  - tmppath: "./tmp/"
  - crtpath: "{{ tmppath }}{{ dnsname }}.crt"
  - pempath: "{{ tmppath }}{{ dnsname }}.pem"
  - csrpath: "{{ tmppath }}{{ dnsname }}.csr"
  - pfxpath: "{{ tmppath }}{{ dnsname }}.pfx"
  - private_key_password: "password"
  tasks:
  - file:
      path: "{{ tmppath }}"
      state: absent
  - file:
      path: "{{ tmppath }}"
      state: directory
  - name: "Generate the private key file to sign the CSR"
    openssl_privatekey:
      path: "{{ pempath }}"
      passphrase: "{{ private_key_password }}"
      cipher: aes256
  - name: "Generate the CSR file signed with the private key"
    openssl_csr:
      path: "{{ csrpath }}"
      privatekey_path: "{{ pempath }}"
      privatekey_passphrase: "{{ private_key_password }}"
      common_name: "{{ dnsname }}"
  - name: "Sign the CSR file as a CA to turn it into a certificate"
    openssl_certificate:
      path: "{{ crtpath }}"
      privatekey_path: "{{ pempath }}"
      privatekey_passphrase: "{{ private_key_password }}"
      csr_path: "{{ csrpath }}"
      provider: selfsigned
  - name: "Convert the signed certificate into a PKCS12 file with the attached private key"
    openssl_pkcs12:
      action: export
      path: "{{ pfxpath }}"
      name: "{{ dnsname }}"
      privatekey_path: "{{ pempath }}"
      privatekey_passphrase: "{{ private_key_password }}"
      passphrase: password
      certificate_path: "{{ crtpath }}"
      state: present

Ansible Behaviour Change

For those of you who are working with #Ansible… Ansible 2.5 is out, and has an unusual documentation change around a key Ansible concept – `with_` loops Where you previously had:

with_dict: "{{ your_fact }}"
or
with_subelements:
- "{{ your_fact }}"
- some_subkey

This now should be written like this:

loop: "{{ lookup('dict', your_fact) }}"
and
loop: "{{ lookup('subelements', your_fact, 'some_subkey') }}"

Fear not, I hear you say, It’s fine, of course the documentation suggests that this is “how it’s always been”…… HA HA HA Nope. This behaviour is new as of 2.5, and needs ansible to be updated to the latest version. As far as I can tell, there’s no way to indicate to Ansible “Oh, BTW, this needs to be running on 2.5 or later”… so I wrote a role that does that for you.

ansible-galaxy install JonTheNiceGuy.version-check

You’re welcome :)

More useful URLs:

Defining Networks with Ansible

In my day job, I’m using Ansible to provision networks in OpenStack. One of the complaints I’ve had about the way I now define them is that the person implementing the network has to spell out all the network elements – the subnet size, DHCP pool, the addresses of the firewalls and names of those items. This works for a manual implementation process, but is seriously broken when you try to hand that over to someone else to implement. Most people just want something which says “Here is the network I want to implement – 192.0.2.0/24″… and let the system make it for you.

So, I wrote some code to make that happen. It’s not perfect, and it’s not what’s in production (we have lots more things I need to add for that!) but it should do OK with an IPv4 network.

Hope this makes sense!

---
- hosts: localhost
  vars:
  - networks:
      # Defined as a subnet with specific router and firewall addressing
      external:
        subnet: "192.0.2.0/24"
        firewall: "192.0.2.1"
        router: "192.0.2.254"
      # Defined as an IP address and CIDR prefix, rather than a proper network address and CIDR prefix
      internal_1:
        subnet: "198.51.100.64/24"
      # A valid smaller network and CIDR prefix
      internal_2:
        subnet: "203.0.113.0/27"
      # A tiny CIDR network
      internal_3:
        subnet: "203.0.113.64/30"
      # These two CIDR networks are unusable for this environment
      internal_4:
        subnet: "203.0.113.128/31"
      internal_5:
        subnet: "203.0.113.192/32"
      # A massive CIDR network
      internal_6:
        subnet: "10.0.0.0/8"
  tasks:
  # Based on https://stackoverflow.com/a/47631963/5738 with serious help from mgedmin and apollo13 via #ansible on Freenode
  - name: Add router and firewall addressing for CIDR prefixes < 30     set_fact:       networks: >
        {{ networks | default({}) | combine(
          {item.key: {
            'subnet': item.value.subnet | ipv4('network'),
            'router': item.value.router | default((( item.value.subnet | ipv4('network') | ipv4('int') ) + 1) | ipv4),
            'firewall': item.value.firewall | default((( item.value.subnet | ipv4('broadcast') | ipv4('int') ) - 1) | ipv4),
            'dhcp_start': item.value.dhcp_start | default((( item.value.subnet | ipv4('network') | ipv4('int') ) + 2) | ipv4),
            'dhcp_end': item.value.dhcp_end | default((( item.value.subnet | ipv4('broadcast') | ipv4('int') ) - 2) | ipv4)
          }
        }) }}
    with_dict: "{{ networks }}"
    when: item.value.subnet | ipv4('prefix') < 30   - name: Add router and firewall addressing for CIDR prefixes = 30     set_fact:       networks: >
        {{ networks | default({}) | combine(
          {item.key: {
            'subnet': item.value.subnet | ipv4('network'),
            'router': item.value.router | default((( item.value.subnet | ipv4('network') | ipv4('int') ) + 1) | ipv4),
            'firewall': item.value.firewall | default((( item.value.subnet | ipv4('broadcast') | ipv4('int') ) - 1) | ipv4)
          }
        }) }}
    with_dict: "{{ networks }}"
    when: item.value.subnet | ipv4('prefix') == 30
  - debug:
      var: networks

Using inspec to test your ansible

Over the past few days I’ve been binge listening to the Arrested Devops podcast. In one of the recent episodes (“Career Change Into DevOps With Michael Hedgpeth, Annie Hedgpeth, And Megan Bohl (ADO102)“) one of the interviewees mentions that she got started in DevOps by using Inspec.

Essentially, inspec is a way of explaining “this is what my server must look like”, so you can then test these statements against a built machine… effectively letting you unit test your provisioning scripts.

I’ve already built a fair bit of my current personal project using Ansible, so I wasn’t exactly keen to re-write everything from scratch, but it did make me think that maybe I should have a common set of tests to see how close my server was to the hardening “Benchmark” guides from CIS… and that’s pretty easy to script in inspec, particularly as the tests in those documents list the “how to test” and “how to remediate” commands to execute.

These are in the process of being drawn up (so far, all I have is an inspec test saying “confirm you’re running on Ubuntu 16.04″… not very complex!!) but, from the looks of things, the following playbook would work relatively well!

---
- name: Make /testing path
  file:
    state: directory
    path: /testing
    owner: root
    group: root
- name: Copy tests to /testing
  copy:
    src: ../files/
    dest: /testing/
    owner: root
    group: root
- name: Ensure ruby is installed
  apt:
    name: "{{ item }}"
    state: present
  with_items:
  - ruby
  - ruby-dev
  - build-essential
  - libffi-dev
- name: Ensure inspec is installed
  gem:
    name: inspec
    state: present
    user_install: no
- name: Run inspec tests
  command: inspec exec /testing

A brief guide to using vagrant-aws

CCHits was recently asked to move it’s media to another host, and while we were doing that we noticed that many of the Monthly shows were broken in one way or another…

Cue a massive rebuild attempt!

We already have a “ShowRunner” script, which we use with a simple Vagrant machine, and I knew you can use other hypervisor “providers”, and I used to use AWS to build the shows, so why not wrap the two parts together?

Firstly, I installed the vagrant-aws plugin:

vagrant plugin install vagrant-aws

Next I amended my Vagrantfile with the vagrant-aws values mentioned in the plugin readme:

Vagrant.configure(2) do |config|
    config.vm.provider :aws do |aws, override|
    config.vm.box = "ShowMaker"
    aws.tags = { 'Name' => 'ShowMaker' }
    config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
    
    # AWS Credentials:
    aws.access_key_id = "DECAFBADDECAFBADDECAF"
    aws.secret_access_key = "DeadBeef1234567890+AbcdeFghijKlmnopqrstu"
    aws.keypair_name = "TheNameOfYourSSHKeyInTheEC2ManagementPortal"
    
    # AWS Location:
    aws.region = "us-east-1"
    aws.region_config "us-east-1", :ami => "ami-c29e1cb8" # If you pick another region, use the relevant AMI for that region
    aws.instance_type = "t2.micro" # Scale accordingly
    aws.security_groups = [ "sg-1234567" ] # Note this *MUST* be an SG ID not the name
    aws.subnet_id = "subnet-decafbad" # Pick one subnet from https://console.aws.amazon.com/vpc/home
    
    # AWS Storage:
    aws.block_device_mapping = [{
      'DeviceName' => "/dev/sda1",
      'Ebs.VolumeSize' => 8, # Size in GB
      'Ebs.DeleteOnTermination' => true,
      'Ebs.VolumeType' => "GP2", # General performance - you might want something faster
    }]
    
    # SSH:
    override.ssh.username = "ubuntu"
    override.ssh.private_key_path = "/home/youruser/.ssh/id_rsa" # or the SSH key you've generated
    
    # /vagrant directory - thanks to https://github.com/hashicorp/vagrant/issues/5401
    override.nfs.functional = false # It tries to use NFS - use RSYNC instead
  end
  config.vm.box = "ubuntu/trusty64"
  config.vm.provision "shell", path: "./run_setup.sh"
  config.vm.provision "shell", run: "always", path: "./run_showmaker.sh"
end

Of course, if you try to put this into your Github repo, it’s going to get pillaged and you’ll be spending lots of money on monero mining very quickly… so instead, I spotted this which you can do to separate out your credentials:

At the top of the Vagrantfile, add these two lines:

require_relative 'settings_aws.rb'
include SettingsAws

Then, replace the lines where you specify a “secret”, like this:

    aws.access_key_id = AWS_ACCESS_KEY
    aws.secret_access_key = AWS_SECRET_KEY

Lastly, create a file “settings_aws.rb” in the same path as your Vagrantfile, that looks like this:

module SettingsAws
    AWS_ACCESS_KEY = "DECAFBADDECAFBADDECAF"
    AWS_SECRET_KEY = "DeadBeef1234567890+AbcdeFghijKlmnopqrstu"
end

This file then can be omitted from your git repository using a .gitignore file.

Running Streisand to provide VPN services on my home server

A few months ago I was a guest on The Ubuntu Podcast, where I mentioned that I use Streisand to terminate my VPN connections. I waffled and blathered a bit about how I set it up, but in the end it comes down to this:

  1. Install Virtualbox on my Ubuntu server. Include the “Ext Pack”.
  2. Install Vagrant on my Ubuntu server.
  3. Clone the Streisand Github repository to my Ubuntu server.
  4. Enter that cloned repository, and edit the Vagrantfile as follows:
    1. Add the line “config.vm.boot_timeout = 65535” after the one starting “config.vm.box”.
    2. Change the streisand.vm.hostname line to be an appropriate hostname for my network, and add on the following line (replace “eth0” with the attached interface on your network and “192.0.2.1” with an unallocated static IP address from your network):
      streisand.vm.network "public_network", bridge: "eth0", ip: "192.0.2.1", :use_dhcp_assigned_default_route => false
    3. Add a “routing” line, as follows (replace 192.0.2.254 with your router IP address):
      streisand.vm.provision "shell", run: "always", inline: "ip route add 0.0.0.0/1 via 192.0.2.254 ; ip route add 128.0.0.0/1 via 192.0.2.254"
    4. Comment out the line “streisand_client_test => true”
    5. Amend the line “streisand_ipv4_address” to reflect the IP address you’ve put above in 4.2.
    6. Remove the block starting “config.vm.define streisand-client do |client|”
  5. Run “vagrant up” in that directory to start the virtual machine. Once it’s finished starting, there will be a folder called “Generated Docs” – open the .html file to see what credentials you must use to access the server. Follow it’s instructions.
  6. Once it’s completed, you should open ports on your router to the IP address you’ve specified. Typically, at least, UDP/500 and UDP/4500 for the IPsec service, UDP/636 for the OpenVPN service and TCP/4443 for the OpenConnect service.

Running Google MusicManager for two profiles

I’ve previously made mention of my addiction to Google Play Music… but I was called out recently, and asked about the script I used at the time. I’m sorry to say that I have had some issues with it, and instead, have resorted to using X forwarding. Here’s how I do it.

I create a user account for that other person (note, GMM will only let you upload to 3 accounts using this method. For any more, you’ll need a virtual machine!).

I then create an SSH public/private key with no passphrase.

ssh-keygen -b 2048 -N “” -C “$(whoami)@localhost” -f ~/.ssh/gmm.id_rsa

I write the public key into that new user’s .ssh/authorized_keys, by running:

ssh-copy-id -i ~/.ssh/gmm.id_rsa bloggsf@localhost

I will be prompted for the password of that account.

Finally, I create this script:

#!/bin/bash
while ! ping -c 1 8.8.8.8 2>/dev/null >/dev/null ; do
  echo Waiting for network...
done

ssh -X bloggsf@localhost -i ~/.ssh/gmm.id_rsa /opt/google/musicmanager/google-musicmanager

This is then added to the startup tasks of my headless-but-running-a-desktop machine.

Use your Debian System with as an iBeacon for Home Automation

I have been using the Home-Assistant application at home to experiment with Home Automation.

One thing I’ve found is that the Raspberry Pi is perfect for a few of the monitoring things that I wanted it to do (see also https://github.com/JonTheNiceGuy/home-assistant-configs for more details of what I’m doing there!).

I’m using the OwnTracks application to talk to an MQTT server, but I could also do with it knowing where I am in the house, so I looked around for some details on iBeacons.

iBeacon is an Apple standard, but it’s very easy to configure on Linux systems. I took some pointers from this article and wrote up a script to turn on the iBeacon on my Raspbian Raspberry Pi 3.

Configuring the Script

When you first run it as root, it will pre-populate a config file in /etc/iBeacon.conf. Edit it and run the script again.

Running the script

This script needs to be run as root, so to test it, or to reconfigure the beacon, run sudo /root/iBeacon.sh (or wherever you put it!)

Making it persistent

To be honest, at this point, I’d probably just stick this into my root Crontab file by adding this line:

@reboot /root/iBeacon.sh | logger

Again, replace /root/iBeacon.sh with wherever you put it!

Please visit this link to see the script and make suggestions on improvements.