Today I learned… Cloud-init doesn’t like you repeating the same things

Because of templates I was building in my post “Today I learned… Ansible Include Templates”, I thought you could repeat the same sections over again. Here’s a snippet of something like what I’d built (after combining lots of templates together):

Note this is a non-working code sample!


#cloud-config
packages:
- iperf
- git

write_files:
- content: {% include 'files/public_key.j2' %}
  path: /root/.ssh/authorized_keys
  owner: root:root
  permission: '0600'
- content: {% include 'files/private_key.j2' %}
  path: /root/.ssh/id_rsa
  owner: root:root
  permission: '0600'

packages:
- byobu

write_files:
- content: |
    #!/bin/bash
    git clone {{ test_scripts }} /root/iperf_scripts
    bash /root/iperf_scripts/run_test.sh
  path: /root/run_test
  owner: root:root
  permission: '0700'

runcmd:
- /root/run_test

I’d get *bits* of it to run – basically, the last file, the last package and the last runcmd… but not all of it.

Turns out, cloud-init doesn’t like having to rebuild all the fragments together. Instead, you need to put them all together, so the write_files items, and the packages items all live in the same area.

Which, when you think about what it’s doing, which is that the parent lines are defining a variable called… well, whatever that line is, and if you replace it, it’s only going to keep the last one, then it all makes sense really!

Today I learned… that you can look at the “cloud-init” files on your target server…

Today I have been debugging why my Cloud-init scripts weren’t triggering on my Openstack environment.

I realised that something was wrong when I tried to use the noVNC console[1] with a password I’d set… no luck. So, next I ran a command to review the console logs[2], and saw a message (now, sadly, long gone – so I can’t even include it here!) suggesting there was an issue parsing my YAML file. Uh oh!

I’m using Ansible’s os_server module, and using templates to complete the userdata field, which in turn gets populated as cloud-init scripts…. and so clearly I had two ways to debug this – prefix my ansible playbook with a few debug commands, but then that can get messy… OR SSH into the box, and look through the logs. I knew I could SSH in, so the cloud-init had partially fired, but it just wasn’t parsing what I’d submitted. I had a quick look around, and found a post which mentioned debugging cloud-init. This mentioned that there’s a path (/var/lib/cloud/instances/$UUID/) you can mess around in, to remove some files to “fool” cloud-init into thinking it’s not been run… but, I reasoned, why not just see what’s there.

And in there, was the motherlode – user-data.txt…. bingo.

In the jinja2 template I was using to populate the userdata, I’d referenced another file, again using a template. It looks like that template needs an extra line at the end, otherwise, it all runs together.

Whew!

This does concern me a little, as I had previously been using this stanza to “simply” change the default user password to something a little less complicated:


#cloud-config
ssh_pwauth: True
chpasswd:
  list: |
    ubuntu:{{ default_password }}
  expire: False

But now that I look at the documentation, I realise you can also specify that as a pre-hashed value (in which case, you would suffix that default_password item above with |password_hash('sha512')) which makes it all better again!

[1] If you run openstack --os-cloud cloud_a console url show servername gives you a URL to visit that has an HTML5 based VNC-ish client. Note the “cloud_a” and “servername” should be replaced by your clouds.yml reference and the server name or server ID you want to connect to.
[2] Like before, openstack --os-cloud cloud_a console log show servername gives you the output of the boot sequence (e.g. dmesg plus the normal startup commands, and finally, cloud-init). It can be useful. Equally, it’s logs… which means there’s a lot to wade through!

Today I learned… Ansible Include Templates

I am building Openstack Servers with the ansible os_server module. One of these fields will accept a very long string (userdata). Typically, I end up with a giant blob of unreadable build script in this field…

Today I learned that I can use this:

---
- name: "Create Server"
  os_server:
    name: "{{ item.value.name }}"
    state: present
    availability_zone: "{{ item.value.az.name }}"
    flavor: "{{ item.value.flavor }}"
    key_name: "{{ item.value.az.keypair }}"
    nics: "[{%- for nw in item.value.ports -%}{'port-name': '{{ ProjectPrefix }}{{ item.value.name }}-Port-{{nw.network.name}}'}{%- if not loop.last -%}, {%- endif -%} {%- endfor -%}]" # Ignore this line - it's complicated for a reason
    boot_volume: "{{ ProjectPrefix }}{{ item.value.name }}-OS-Volume" # Ignore this line also :)
    terminate_volume: yes
    volumes: "{%- if item.value.log_size is defined -%}[{{ ProjectPrefix }}{{ item.value.name }}-Log-Volume]{%- else -%}{{ omit }}{%- endif -%}"
    userdata: "{% include 'templates/userdata.j2' %}"
    auto_ip: no
    timeout: 65535
    cloud: "{{ cloud }}"
  with_dict: "{{ Servers }}"

This file (/path/to/ansible/playbooks/servers.yml) is referenced by my play.yml (/path/to/ansible/play.yml) via an include, so the template reference there is in my templates directory (/path/to/ansible/templates/userdata.j2).

That template can also then reference other template files itself (using {% include 'templates/some_other_file.extension' %}) so you can have nicely complex userdata fields with loads and loads of detail, and not make the actual play complicated (or at least, no more than it already needs to be!)

Using Python-OpenstackClient and Ansible with K5

Recently, I have used K5, which is an instance of OpenStack, run by Fujitsu (my employer). To do some of the automation tasks I have played with both python-openstackclient and Ansible. This post is going to cover how to get those tools to work with K5.

I have access to a Linux virtual machine (Ubuntu 16.04) and the Windows Subsystem for Linux in Windows 10 to run “Bash on Ubuntu on Windows”, and both accept the same set of commands.

In order to run these commands, you need a couple of dependencies. Your mileage might vary with other Linux distributions, but, for Ubuntu based distributions, run this command:

sudo apt install python-pip build-essential libssl-dev libffi-dev python-dev

Next, use pip to install the python modules you need:

sudo -H pip install shade==1.11.1 ansible cryptography python-openstackclient

If you’re only ever going to be working with a single project, you can define a handful of environment variables prefixed OS_, like this:

export OS_USERNAME=BloggsF
export OS_PASSWORD=MySuperSecretPasswordIsHere
export OS_REGION_NAME=uk-1
export OS_USER_DOMAIN_NAME=YourProjectName
export OS_PROJECT_NAME=YourProjectName-prj
export OS_PROJECT_ID=baddecafbaddecafbaddecafbaddecaf
export OS_AUTH_URL=https://identity.uk-1.cloud.global.fujitsu.com/v3
export OS_VOLUME_API_VERSION=2
export OS_IDENTITY_API_VERSION=3

But, if you’re working with a few projects, it’s probably worth separating these out into clouds.yml files. This would be stored in ~/.config/openstack/clouds.yml with the credentials for the environment you’re using:

---
clouds:
  root:
    identity_api_version: 3
    regions:
    - uk-1
    auth:
      auth_url: https://identity.uk-1.cloud.global.fujitsu.com/v3
      password: MySuperSecretPasswordIsHere
      project_id: baddecafbaddecafbaddecafbaddecaf
      project_name: YourProjectName-prj
      username: BloggsF
      user_domain_name: YourProjectName

Optionally, you can separate out the password, username or any other “sensitive” information into a secure.yml file stored in the same location (removing those lines from the clouds.yml file), like this:

---
clouds:
  root:
    auth:
      password: MySuperSecretPasswordIsHere

Now, you can use the Python based Openstack Client, using this invocation:

openstack --os-cloud root server list

Alternatively you can use the Ansible Openstack (and K5) modules, like this:

---
tasks:
- name: "Authenticate to K5"
  k5_auth:
    cloud: root
  register: k5_auth_reg
- name: "Create Network"
  k5_create_network:
    name: "Public"
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Create Subnet"
  k5_create_subnet:
    name: "Public"
    network_name: "Public"
    cidr: "192.0.2.0/24"
    gateway_ip: "192.0.2.1"
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Create Router"
  k5_create_router:
    name: "Public"
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Attach private network to router"
  os_router:
    name: "Public"
    state: present
    network: "inf_az1_ext-net02"
    interfaces: "Public"
    cloud: root
- name: "Create Servers"
  os_server:
    name: "Server"
    availability_zone: "uk-1a"
    flavor: "P-1"
    state: present
    key_name: "MyFirstKey"
    network: "Public-Network"
    image: "Ubuntu Server 14.04 LTS (English) 02"
    boot_from_volume: yes
    terminate_volume: yes
    security_groups: "Default"
    auto_ip: no
    timeout: 7200
    cloud: root

Working with complicated template data UserData in Ansible

My new job means I’m currently building a lot of test boxes with Ansible, particularly OpenStack guests. This means I’m trying to script as much as possible without actually … getting my hands dirty with the actual “logging into it and running things” perspective.

This week, I hit a problem standing up a popular firewall vendor’s machine with Ansible, because I was trying to bypass the first-time-wizard… anyway, it wasn’t working, and I couldn’t figure out why. I talked to my colleague [mohclips] and he eventually told me that I needed to use a template, because what I was trying to do was too complicated.

But, damn him, I knew that wasn’t the answer :)

Anyway, I found this comment on a ticket, which lead me to the following… if you’re finding that your userdata: variable in the os_server module of Ansible isn’t working, you might need to wrap it up like this:

userdata: |
  {%- raw -%}#!/bin/bash
  # Kill script if the pipe fails
  set -euf -o pipefail
  # Write everything from this point on to Syslog
  echo " == Set admin credentials == "
  clish -c 'set user admin password-hash {% endraw -%}{{ default_password|password_hash('sha512') }}{%- raw -%}' -s
  {% endraw %}

Note that, if you have a space before your variable, use {% endraw -%} and if you’ve a space after it, use {%- raw %} as the hyphen means “ditch all the spaces before/after this command”.

Using Expect to SFTP a file

Because of technical limitations on a pair of platforms I’m using at work, I am unable to set-up key-based SFTP or SCP to transfer files between the pair of them, so I knocked together this short script using the TCL based Expect language.


#!/usr/bin/expect
set arg1 [lindex $argv 0]
set arg2 [lindex $argv 1]
set arg3 [lindex $argv 2]
set timeout 1000
spawn sftp "$arg2"
expect {
yes {
send "yes\r"
exp_continue
}
ass {
send "$arg3\r"
exp_continue
}
sftp {
send "put $arg1\r"
expect {
100% {
send "quit\r"
exp_continue
}
}
}
}

view raw

upload.exp

hosted with ❤ by GitHub

There’s no error checking here, which isn’t great, but as a quick-and-dirty script to SFTP files to a box which needs the password each run… it works! :)

GPG Encrypting files using a keyserver

Another “at work” post!

I’ve been generating files which need to be distributed via a file server, but need to be encrypted using GPG (the open source PGP application). Rather than managing keys for a large number of users, instead, I have a text file with the user names in, and a batch file. Please see the below gist for details :)

Installing Symantec Endpoint Protection (SEP) on Ubuntu 14.04

At work we use Symantec Endpoint Protection, and in a lab, I was asked to confirm whether we could install it on our Ubuntu 14.04 servers. This took a few hops to get it installed, so I figured, I’d publish how I got it working, to save some other poor soul the trouble :)

Firstly, add the webupd8team’s Java PPA and update the repository cache: sudo add-apt-repository ppa:webupd8team/java && sudo apt-get update

This gives you the ability to install the Java 8 installer: sudo apt-get install oracle-java8-installer

This should download the install files, but for some reason, I was struggling to download it (the install script seems to struggle with downloading the actual .tar.gz file from Oracle), so I manually followed the link to http://download.oracle.com/otn-pub/java/jdk/8u77-b03/jdk-8u77-linux-x64.tar.gz, accepted the license, and placed the file in /var/cache/oracle-jdk8-installer/ and then re-ran the above apt-get install line.

— Note: This above issue was because I was running a caching proxy, which somehow doesn’t play nicely with this script. Turn off your proxy – should be all good :)

Next I had to install the Java Cryptography Extension which I got from the Java SE page. I placed this file in /tmp/jce_policy-8.zip (the filename is the one Oracle use) and replaced the files in /usr/lib/jvm/java-8-oracle/jre/lib/security with the ones from the extracted archive with this line: cp -b /tmp/UnlimitedJCEPolicyJDK8/*.jar /usr/lib/jvm/java-8-oracle/jre/lib/security.

The SEP client also has a dependency on the 32bit version of GLibc. I installed this with sudo apt-get install libc6-i386

I was then, finally, able to install the SEP client by unpacking the installer zip file, and running sudo bash install.sh -i from the path I’d unpacked the zip file in.

Not very complicated, I guess!

— Sources:

  • https://ubuntuincident.wordpress.com/2011/04/14/install-the-java-cryptography-extension-jce/
  • http://www.linuxquestions.org/questions/linux-newbie-8/how-to-install-32-bit-glibc-2-9-or-later-on-64-bit-ubuntu-12-04-a-4175413667/
  • http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html

Stripping a UK O2 Samsung Galaxy SIII Mini down to the bare essentials

The company I work for have recently issued all On Call engineers in my team a Samsung Galaxy SIII Mini to give us access to company e-mail and resources out of hours. Rather than shipping a customized image, we have received a stock O2 imaged mobile, and so this is my limited guide to bringing this to as close to “Stock” Android as I can manage (or want).

Most of what we need is provided to us using a commercial solution called Touchdown, so I won’t be covering that here, as whatever you get shipped to you will not include that. I’ve elected not to use the device for my personal systems, barring my Google calendar, which means I’ll stand a fighting chance of not booking overtime and other work things for personal appointments.

So, on power-on, I completed the post-install steps, including setting up my Google account. I decided not to keep the device in sync with my Google account, as I already have a few other Android devices, and I don’t want to get my work infrastructure mixed up with my home kit.

Next, I went into Settings, and from there into the Google Account. I clicked on my e-mail address and unselected the following options:

  • Sync App Data
  • Sync Contacts
  • Sync Gmail
  • Sync Internet
  • Sync People details

After that, I went into Application Manager (again, in Settings), and swapped to the “All” tab. Firstly, I needed to clear out the downloaded contacts, which I did by selecting Contacts Storage, and then pressing the “Clear Data” button.

Next, I disabled all the applications that I either don’t need, or don’t want on my work phone. I did this by selecting each in turn, and then selecting the option to disable them. Here’s the list:

  • Amazon MP3
  • eBay
  • Flipboard
  • Gallery
  • Game Hub
  • Gmail
  • Google Play Books
  • Google Play Magazines
  • Google Play Music
  • Google+
  • Music
  • O2 Space
  • S Planner
  • S Planner Widget
  • S Suggest
  • S Voice
  • Samsung Account
  • Samsung Apps
  • Samsung Backup Provider
  • Samsung Browser SyncAdapter
  • Samsung Calendar SyncAdapter
  • Samsung Cloud Data Relay
  • Samsung Contact SyncAdapter
  • Samsung Push Service
  • Samsung Syncadapters
  • Tags
  • Talk
  • Talkback
  • Video Hub
  • Yahoo! Finance
  • Yahoo! News
  • YouTube

Wow, isn’t that a list!

My next step was to hide some of the applications I don’t need. To do this, I went into the applications page, pressed the menu button, and selected “Hide applications”. This puts selection boxes next to all the applications on the page, and once you’ve done selecting options, press “Done” in the top right corner to hide them. Here’s my list:

  • Contacts
  • Downloads
  • E-Mail
  • FM Radio
  • Google Settings
  • Help
  • Memo
  • Music Player
  • My Files
  • Video Player
  • Voice Recorder
  • Voice Search

Lastly, installed a couple of applications from the Play Store:

Once I’d got Agenda Widget Plus, and Google Keyboard configured, I hid those applications in the applications pane too.

After all of that, I set up Touchdown… which you’ll need to follow up though your own instructions!

One final thing before I wrap this all up… even though I’m on-call, this doesn’t include being engaged via e-mail. As such, my e-mail doesn’t need to disturb me, and so I’ve disabled the touchdown application’s notifications for e-mail. To do this, go into Touchdown, make sure you’re at the “main” screen (not the default e-mail screen, but the one which also includes all your tasks and calendar options), and then press the menu button, press “Settings”, and select the “Advanced” tab. Scroll right to the bottom of the list, and press the “Email Alerts” button. Select “Customize settings” and then select appropriate options. If you leave nothing ticked, all you’ll get is a flag in the notifications tray showing an e-mail has appeared. Personally, I’ve turned on “Enable lights” and picked a colour, so I can quickly see whether I’ve had a mail just by checking the screen.