Today I learned… Ansible Include Templates

I am building Openstack Servers with the ansible os_server module. One of these fields will accept a very long string (userdata). Typically, I end up with a giant blob of unreadable build script in this field…

Today I learned that I can use this:

- name: "Create Server"
    name: "{{ }}"
    state: present
    availability_zone: "{{ }}"
    flavor: "{{ item.value.flavor }}"
    key_name: "{{ }}"
    nics: "[{%- for nw in item.value.ports -%}{'port-name': '{{ ProjectPrefix }}{{ }}-Port-{{}}'}{%- if not loop.last -%}, {%- endif -%} {%- endfor -%}]" # Ignore this line - it's complicated for a reason
    boot_volume: "{{ ProjectPrefix }}{{ }}-OS-Volume" # Ignore this line also :)
    terminate_volume: yes
    volumes: "{%- if item.value.log_size is defined -%}[{{ ProjectPrefix }}{{ }}-Log-Volume]{%- else -%}{{ omit }}{%- endif -%}"
    userdata: "{% include 'templates/userdata.j2' %}"
    auto_ip: no
    timeout: 65535
    cloud: "{{ cloud }}"
  with_dict: "{{ Servers }}"

This file (/path/to/ansible/playbooks/servers.yml) is referenced by my play.yml (/path/to/ansible/play.yml) via an include, so the template reference there is in my templates directory (/path/to/ansible/templates/userdata.j2).

That template can also then reference other template files itself (using {% include 'templates/some_other_file.extension' %}) so you can have nicely complex userdata fields with loads and loads of detail, and not make the actual play complicated (or at least, no more than it already needs to be!)

Using Python-OpenstackClient and Ansible with K5

Recently, I have used K5, which is an instance of OpenStack, run by Fujitsu (my employer). To do some of the automation tasks I have played with both python-openstackclient and Ansible. This post is going to cover how to get those tools to work with K5.

I have access to a Linux virtual machine (Ubuntu 16.04) and the Windows Subsystem for Linux in Windows 10 to run “Bash on Ubuntu on Windows”, and both accept the same set of commands.

In order to run these commands, you need a couple of dependencies. Your mileage might vary with other Linux distributions, but, for Ubuntu based distributions, run this command:

sudo apt install python-pip build-essential libssl-dev libffi-dev python-dev

Next, use pip to install the python modules you need:

sudo -H pip install shade==1.11.1 ansible cryptography python-openstackclient

If you’re only ever going to be working with a single project, you can define a handful of environment variables prefixed OS_, like this:

export OS_USERNAME=BloggsF
export OS_PASSWORD=MySuperSecretPasswordIsHere
export OS_REGION_NAME=uk-1
export OS_USER_DOMAIN_NAME=YourProjectName
export OS_PROJECT_NAME=YourProjectName-prj
export OS_PROJECT_ID=baddecafbaddecafbaddecafbaddecaf
export OS_AUTH_URL=

But, if you’re working with a few projects, it’s probably worth separating these out into clouds.yml files. This would be stored in ~/.config/openstack/clouds.yml with the credentials for the environment you’re using:

    identity_api_version: 3
    - uk-1
      password: MySuperSecretPasswordIsHere
      project_id: baddecafbaddecafbaddecafbaddecaf
      project_name: YourProjectName-prj
      username: BloggsF
      user_domain_name: YourProjectName

Optionally, you can separate out the password, username or any other “sensitive” information into a secure.yml file stored in the same location (removing those lines from the clouds.yml file), like this:

      password: MySuperSecretPasswordIsHere

Now, you can use the Python based Openstack Client, using this invocation:

openstack --os-cloud root server list

Alternatively you can use the Ansible Openstack (and K5) modules, like this:

- name: "Authenticate to K5"
    cloud: root
  register: k5_auth_reg
- name: "Create Network"
    name: "Public"
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Create Subnet"
    name: "Public"
    network_name: "Public"
    cidr: ""
    gateway_ip: ""
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Create Router"
    name: "Public"
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Attach private network to router"
    name: "Public"
    state: present
    network: "inf_az1_ext-net02"
    interfaces: "Public"
    cloud: root
- name: "Create Servers"
    name: "Server"
    availability_zone: "uk-1a"
    flavor: "P-1"
    state: present
    key_name: "MyFirstKey"
    network: "Public-Network"
    image: "Ubuntu Server 14.04 LTS (English) 02"
    boot_from_volume: yes
    terminate_volume: yes
    security_groups: "Default"
    auto_ip: no
    timeout: 7200
    cloud: root

One to read or watch: “Programming is Forgetting: Toward a New Hacker Ethic”

Here is a transcript of a talk by Allison Parrish at the Open Hardware Summit in Portland, OR. The talk “Programming is Forgetting: Toward a New Hacker Ethic” is a discussion about the failings of the book “Hackers” by Steven Levy. Essentially, that book proposed (in the 80’s) a set of ethics for Hackers (which is to say, creative programmers or engineers, not malicious operators). Allison suggests that many of the parables in the book do not truly reflect the “Hacker Ethic”, and revises them for today’s world.

Her new questions (not statements) are as follows:

  • Who gets to use what I make? Who am I leaving out? How does what I make facilitate or hinder access?
  • What data am I using? Whose labor produced it and what biases and assumptions are built into it? Why choose this particular phenomenon for digitization or transcription? And what do the data leave out?
  • What systems of authority am I enacting through what I make? What systems of support do I rely on? How does what I make support other people?
  • What kind of community am I assuming? What community do I invite through what I make? How are my own personal values reflected in what I make?

This is a significant re-work of the original “Hacker Ethic“, and you should really either watch or read the talk to see how she got to these from the original, especially as it’s not as punchy as the original.

I’d like to think I was thinking of things like these questions when I wrote CampFireManager and CCHits.

Development Environment Replication with Vagrant and Puppet

This week, I was fortunate enough to meet up with the Cheadle Geeks group. I got talking to a couple of people about Vagrant and Puppet, and explaining how it works, and I thought the best thing to do would be to also write that down here, so that I can point anyone who missed any of what I was saying to it.

Essentially, Vagrant is program to read a config file which defines how to initialize a pre-built virtual machine. It has several virtual machine engines which it can invoke (see [1] for more details on that), but the default virtual machine to use is VirtualBox.

To actually find a virtual box to load, there’s a big list over at which have most standard cloud servers available to you. Personally I use the Ubuntu Precise 32bit image from for my open source projects (which means more developers can get involved). Once you’ve picked an image, use the following command to get it installed on your development machine (you only need to do this step once per box!):

vagrant box add {YourBoxName} {BoxURL}

After you’ve done that, you need to set up the Vagrant configuration file.

cd /path/to/your/dev/environment
mkdir Vagrant
cd Vagrant
vagrant init {YourBoxName}

This will create a file called Vagrantfile in /path/to/your/dev/environment/Vagrant. It looks overwhelming at first, but if you trim out some of the notes (and tweak one or two of the lines), you’ll end up with a file which looks a bit like this:

Vagrant.configure("2") do |config| = "{YourBoxName}"
  config.vm.hostname = "{}"
  config.vm.box_url = "{BoxURL}" :forwarded_port, guest: 80, host: 8080
  # :public_network
  config.vm.synced_folder "../web", "/var/www"
  config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "manifests"
    puppet.manifest_file  = "site.pp"

This assumes you’ve replaced anything with {}’s in it with a real value, and that you want to forward TCP/8080 on your machine to TCP/80 on that box (there are other work arounds, using more Vagrant plugins, different network types, or other services such as pagekite, but this will do for now).

Once you’ve got this file, you could start up your machine and get a bare box, but that’s not much use to you, as you’d have to tell people how to configure your development environment every time they started up a new box. Instead, we’ll be using a Provisioning service, and we’re going to use Puppet for that.

Puppet was originally designed as a way of defining configuration across all an estate’s servers, and a lot of tutorials I’ve found online explain how to use it for that, but when we’re setting up Puppet for a development environment, we just need a simple file. This is the site.pp manifest, and in here we define the extra files and packages we need, plus any commands we need to run. So, let’s start with a basic manifest file:

node default {


Wow, isn’t that easy? :) We need some more detail than that though. First, let’s make sure the timezone is set. I live in the UK, so my timezone is “Europe/London”. Let’s put that in. We also need to make sure that any commands we run have the right path in them. So here’s our revised, debian based, manifest file.

node default {
    Exec {
        path => '/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/sbin:/usr/sbin'

    package { "tzdata":
        ensure => "installed"

    file { "/etc/timezone":
        content => "Europe/London\n",
        require => Package["tzdata"]

    exec { "Set Timezone":
        unless => "diff /etc/localtime /usr/share/zoneinfo/`cat /etc/timezone`",
        command => "dpkg-reconfigure -f noninteractive tzdata",
        require => File["/etc/timezone"]

OK, so we’ve got some pretty clear examples of code to run here. The first Exec statement must always be in there, otherwise it gets a bit confused, but after that, we’re making sure the package tzdata is installed, we then make sure that, once the tzdata package is installed, we create or update the /etc/timezone file with the value we want, and then we use the dpkg-reconfigure command to set the timezone, but only if the timezone isn’t already set to that.

Just to be clear, this file describes what the system should look like at the end of it running, not a step-by-step guide to getting it running, so you might find that some of these packages install out of sequence, or something else might run before or after when you were expecting it to run. As a result, you should make good use of the “require” and “unless” statements if you want a proper sequence of events to occur.

Now, so far, all this does is set the timezone for us, it doesn’t set up anything like Apache or MySQL… perhaps you want to install something like WordPress here? Well, let’s see how we get other packages installed.

In the following lines of code, we’ll assume you’re just adding this text above the last curled bracket (the “}” at the end).

First, we need to ensure our packages are up to date:

exec { "Update packages":
    command => "sudo apt-get update && sudo apt-get dist-upgrade -y",

Here’s Apache getting installed:

package { "apache2":
    ensure => "installed",
    require => Exec['Update packages']

And, maybe you’ll want to set up something that needs mod_rewrite and a custom site? Add this to your Vagrantfile

config.vm.synced_folder "../Apache_Site", "/etc/apache2/shared_config"

Create a directory called /path/to/your/dev/environment/Apache_Site which should contain your apache site configuration file called “default”. Then add this to your site.pp

exec { "Enable rewrite":
    command => 'a2enmod rewrite',
    onlyif => 'test ! -e /etc/apache2/mods-enabled/rewrite.load',
    require => Package['apache2']

file { "/etc/apache2/sites-enabled/default":
  ensure => link,
  target => "/etc/apache2/shared_config/default",

So, at the end of all this, we have the following file structure:

+ -- /Apache_Site
|    + -- default
+ -- /web
|    + -- index.html
+ -- /Vagrant
     + -- /manifests
     |    + -- site.pp
     + -- Vagrantfile

And now, you can add all of this to your Git repository [2], and off you go! To bring up your Vagrant machine, type (from the Vagrant directory):

vagrant up

And then to connect into it:

vagrant ssh

And finally to halt it:

vagrant halt

Or if you just want to kill it off…

vagrant destroy

If you’re tweaking the provisioning code, you can run this instead of destroying it and bringing it back up again:

vagrant provision

You can do some funky stuff with running several machines, and using the same puppet file for all of those, but frankly, that’s a topic for another day.

[1] Vagrant is extended using plugins. There is a list of plugins on this Github Wiki Page. The plugins here can include additional virtual machine back ends (called Providers in Vagrant terminology), and methods of configuring the OS after bootup (called Provisioners), but also anything around defining where to find resources, to define network addresses, even to handle caches and proxies.

[2] If you’re not using Git, you should be! However, you might want to add some stuff to your .gitignore – in particular, Vagrant adds a directory called /path/to/your/dev/environment/Vagrant/.vagrant where it puts the VMs it creates.

Building a WPA2 Protected Wireless Access Extender for Jogglers using Ubuntu 12.04

Shesh! What a lot of keywords in the title!

For those who don’t know what some of those key words were, I’ll break down the title

  • Ubuntu is a Linux distribution, and 12.04 is the version number of the latest Long Term Stable version.
  • Joggler is the name of a device sold by O2 a couple of years ago. It is a re-branded OpenPeak tablet.
  • A Wireless Access Extender is a device like a WiFi enabled router, but it uses the same DHCP pool and should use the same SSID name and WPA2 passphrase.
  • WPA2 is the latest incarnation of the WiFi security protocol. It is currently (at this time, as far as I know) uncracked, unlike WPA1 or WEP.

So, now that we know what I’m talking about, let’s look at what components we will be using today.

  • An O2 Joggler. EBay lists them from between £30 and £100. They originally sold for around £100, but got popular when O2 dropped the price to £50. They are no longer available for sale from O2, hence EBay.
  • A wired network connection. I’m using a pair of Ethernet over Power (or “HomePlug”) devices to let me position this device in a useful place in my house. I’ve had a lot of success with the 200M devices sold by, but if I were buying new today, I’d probably stretch up to the 500M devices, as they will be Half Duplex (like a narrow street permitting traffic only one way at a time), and will loose some data due to interference and “collisions” – where two devices on the Ethernet over Power “network” are talking at the same time. Ultimately, you won’t get the equivalent to 100M Full Duplex with the 200M devices, but should do with the 500M devices.
  • A USB stick. This needs to be 4Gb or greater, but not all devices are suitable. I bought some 4Gb sticks from and found they only actually held around 3.5Gb… making them unsuitable. I bought three 8Gb sticks from, but only used one for this task!
  • A Ubuntu 12.04 install. Actually, I used the Xubuntu 12.04 image, because I didn’t need everything that Ubuntu 12.04 gave me. This is a special non-official build of Xubuntu, customised for Joggler hardware and it’s touchscreen, and is what I’ll be moving all my Jogglers in the house to, eventually, however, the principals in making all of this stuff work will apply just as much to Ubuntu as it would Xubuntu – special build or not!
  • Once installed, you’ll use a combination of VNC and SSH to manage your device, these will be through the X11VNC project and OpenSSH-Server. You should have an SSH client (for Linux/Mac, ssh should be fine, for Windows, use PuTTY) and a VNC client (for Ubuntu, I use Remmina, for Windows, I use TightVNC).

So, you’ve got all your goodies, and you’re ready to go. Let’s do this!

  1. Transfer the Xubuntu image to the USB stick. This is a simple task, and is clearly documented on the site where I got the Xubuntu image from, and involves you copying the image directly to the USB stick, not to one of it’s partitions. It sounds complicated, it really isn’t.
  2. Stick the Xubuntu stick into the side of the Joggler. Get used to that shape, as it’s going to be in the side of that from now on. This is because the Linux distribution needs more than the 1Gb that the Joggler holds internally.
  3. Plug in the HomePlug device – make it as close to the wall as you can make it! I’ve had experience of it being three 4way plug strips away from the wall and it worked fine, but I’ve also had the same HomePlug only one 4way away, and it’s completely failed to work, and had to juggle all my sockets to get it plugged directly into the wall. I think it may be down to the number of “noisy” plugs in the same 4way, but I can’t be sure. Just experiment!
  4. Plug your Ethernet cable between the HomePlug and the Joggler.
  5. Power on the Joggler. It will start up with an O2 logo (or possibly an “OpenPeak” logo – depends on when the device was manufactured)  – sometimes either of these may corrupt or show with a big white block as it’s booting. Don’t worry too much about this, we’ll stay away from the boot screen as much as possible! :)
  6. Once you get to a blue screen with icons on it – this is Xubuntu (well, actually XFCE4, but the semantics are moot really). Click on the blue spot in the top left corner of the screen – it may be a little fiddly – and select Ubuntu Software Centre.
  7. Open the “Florence” keyboard – found by pressing the small grid icon near the clock in the top right corner of the screen. If you struggle with this keyboard (I did), you may find it easier to use the “OnBoard” keyboard, found through the applications menu (again, via the blue button in the top corner).
  8. Select the Search box in the Software Centre and search for OpenSSH-Server. Click on the only entry which comes back (you need to search for the exact term) and then click install. While that’s installing, click on the two arrows icon in the top right corner, and select Connection Information. Make a note of the IP address you have received. Once it’s finished installing you can move away to something a little more comfortable to work on your Joggler!
  9. SSH to your Joggler’s IP address – the username for the device is “joggler” and the password is also “joggler”. All of the following you’ll need to be root for. I always use the following line to become root:
    sudo su -
  10. The wireless driver that is installed by default on the Jogglers don’t support “Master” mode – the mode you need to be a wifi access point or extender, so you’ll need to change the wireless driver. Thanks to this post, we know that you edit the file /etc/modprobe.d/joggler.conf and move the comment symbol (#) from before the line blacklist rt2870sta to the line blacklist rt2800usb. It should look like this after you’re done:
    # blacklist rt2800usb
    blacklist rt2870sta
  11. We need to bridge the wlan0 and eth0 interfaces.
    1. Install bridge-utils using apt-get install.
    2. Now we’ll start to configure the bridge. Edit /etc/network/interfaces to create your bridge interfaces.
      auto lo
      iface lo inet loopback
      auto eth0
      iface eth0 inet manual
      auto wlan0
      iface wlan0 inet manual
          pre-up service hostapd start
          post-up brctl addif br0 wlan0
      auto br0
      iface br0 inet dhcp
          bridge_ports eth0 wlan0
          pre-up iptables-restore -c < /etc/iptables.rules
          post-down iptables-save -c > /etc/iptables.rules

      If you want to use a static IP address instead of a DHCP one, then change the last block (auto br0; iface br0 inet dhcp) to the following (this assumes your network is a 192.168.0/24 with .1 as your router to the outside world):

      auto br0
      iface br0 inet static
          bridge_ports eth0 wlan0
    3. Setup /etc/sysctl.conf to permit forwarding of packets. Find, and remove the comment symbol (#) from the line which looks like this:
      # net.ipv4.ip_forward = 1
    4. Create your initial /etc/iptables.rules (this is based on details from this page) and then “restore” them using iptables.
      :INPUT ACCEPT [0:0]
      :FORWARD ACCEPT [0:0]
      :OUTPUT ACCEPT [1:81]
      -A FORWARD -m state --state INVALID -j DROP
      -A FORWARD -i wlan0 -o eth0 -j ACCEPT
      -A FORWARD -i eth0 -o wlan0 -j ACCEPT
    5. Check the iptables have restored properly by running iptables -L -v which should return the following data:
      # iptables -L -v
      Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
       pkts bytes target     prot opt in     out     source               destination         
      Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
       pkts bytes target     prot opt in     out     source               destination
          0     0 ACCEPT     all  --  any    any     anywhere             anywhere             state RELATED,ESTABLISHED
          0     0 DROP       all  --  any    any     anywhere             anywhere             state INVALID
          0     0 ACCEPT     all  --  wlan0  eth0    anywhere             anywhere
          0     0 ACCEPT     all  --  eth0   wlan0   anywhere             anywhere            
      Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
       pkts bytes target     prot opt in     out     source               destination
  12. Now you’ve got a bridged interface, and your wifi adaptor is ready to go, let’s get the DHCP relay in and working right.
    1. apt-get install dhcp3-relay
    2. It’ll ask you where to forward the DHCP requests to – that is your current gateway – if you have your network as with the gateway as .1, then it should be
    3. Next, it’ll ask which interfaces to listen on – this is br0.
    4. The last screen asks for some options to configure – this is “-m forward” (without the quote marks).
  13. Last thing to do, we need to configure something to listen on the wifi interface to provide the Access Point facility to your device. This is “hostapd”.
    1. apt-get install hostapd
    2. zcat /usr/share/doc/hostapd/examples/hostapd.conf.gz > /etc/hostapd/hostapd.conf
    3. Edit /etc/hostapd/hostapd.conf replacing the following config items:
      FROM: # driver = hostapd
      TO:   driver = nl80211
      FROM: #country_code = US
      TO:   country_code = GB
      FROM: hw_mode = a
      TO:   hw_mode = g
      FROM: channel = 60
      TO:   channel = 12
      FROM: #ieee80211n = 1
      TO:   ieee80211n = 1
      FROM: #wpa = 1
      TO:   wpa = 2
      FROM: #wpa_passphrase=secret passphrase
      TO:   wpa_passphrase=MySecretPassword
      FROM: #wpa_pairwise = TKIP CCMP
      TO:   wpa_pairwise = TKIP CCMP
    4. Edit /etc/default/hostapd amending the DAEMON_CONF line to show /etc/hostapd/hostapd.conf

Reboot, and your access point should come to life! Huzzah!! Initially it’ll have the SSID of “test” (it’s in /etc/hostapd/hostapd.conf as the config line “ssid = test”) but you should probably change it to the same SSID as your main router. If you do that, ensure your WPA passphrase is the same as your main router too, otherwise your network will get very confused!

So, now you’ve got an Access extender, running Ubuntu… what else could you do with it? Well, I run one of two things on all of mine – sqeezeplay or vlc monitoring a webcam. All very useful stuff, and stuff I was doing with it before it was an access extender!

Getting started with Unit Testing for PHP

Unit testing seems like a bit of a dark art when you’re first introduced to it. “Create this new file. Tell it what is supposed to be the result when you run a test, and it’ll tell you if you’re right nor not.”

Let’s start with a pseudocode example:

test->assertTrue(1+1 = 2); // Test returns true, huzzah!
test->assertFalse(1+1 = 3); // Test returns false. Those integers must not have been large enough

I want to use PHPUnit, and for me the easiest way to get this and the rest of the tools I’ll be referring to in this collection of posts is to install “The PHP Quality Assurance Toolchain“. On my Ubuntu install, this was done as follows:

sudo pear upgrade PEAR
sudo pear config-set auto_discover 1
sudo pear install --all-deps

Now we’ve got the tools in place, let’s set up the directory structure.

+ -- Classes
|    + -- Config.php
+ -- Tests
     + -- ConfigTest.php

In here, you see we’ve created two files, one contains the class we want to use, and the other contains the tests we will be running.

So, let’s slap on the veneer of coating that these two files need to be valid to test.


class Config



include dirname(__FILE__) . '/../Classes/Config.php';

class ConfigTest extends PHPUnit_Framework_TestCase

So, just to summarise, here we have two, essentially empty classes.

Let’s put some code into the test file.


include dirname(__FILE__) . '/../Classes/Config.php';

class ConfigTest extends PHPUnit_Framework_TestCase
  public function testCreateObject()
    $config = new Config();

We can now run this test from the command line as follows:

phpunit Tests/ConfigTest.php

phpunit Tests/01_ConfigTest.php
PHPUnit 3.6.10 by Sebastian Bergmann.


Time: 1 second, Memory: 3.00Mb

OK (1 test, 1 assertion)

That was nice and straightforward!

Let’s add some more code!

In ConfigTest, let’s tell it to load some configuration, using a config file.


include dirname(__FILE__) . '/../Classes/Config.php';

class ConfigTest extends PHPUnit_Framework_TestCase
  public function testCreateObject()
    $config = new Config();

  public function testLoadConfig()
    $config = new Config();

And now when we run it?

PHP Fatal error:  Call to undefined method Config::load() in /var/www/PhpBetterPractices/Tests/ConfigTest.php on line 16

Ah, perhaps we need to write some code into /Classes/Config.php

class Config
  public function load()
    include dirname(__FILE__) . '/../Config/default_config.php';

But, running this, again, we get an error message!

PHPUnit 3.6.10 by Sebastian Bergmann.


Time: 0 seconds, Memory: 3.00Mb

There was 1 error:

1) ConfigTest::testLoadConfig
include(/var/www/PhpBetterPractices/Config/default_config.php): failed to open stream: No such file or directory


Tests: 2, Assertions: 1, Errors: 1.

So, we actually need to check that the file exists first, perhaps we should throw an error if it doesn’t? We could also pass the name of the config file to pass to the script, which would let us test more and different configuration options, should we need them.

class Config
    public function load($file = null)
        if ($file == null) {
            $file = 'default.config.php';

        $filename = dirname(__FILE__) . '/../Config/' . $file;

        if (file_exists($filename)) {
            include $filename;
        } else {
            throw new InvalidArgumentException("File not found");

So, here’s the new UnitTest code:

class ConfigTest extends PHPUnit_Framework_TestCase
    public function testCreateObject()
        $config = new Config();

    public function testLoadConfig()
        $config = new Config();

     * @expectedException InvalidArgumentException
    public function testFailLoadingConfig()
        $config = new Config();
        @$config->load('A file which does not exist');

This assumes the file /Config/default.config.php exists, albeit as an empty file.

So, let’s run those tests and see what happens?

PHPUnit 3.6.10 by Sebastian Bergmann.


Time: 0 seconds, Memory: 3.25Mb

OK (3 tests, 2 assertions)

Huzzah! That’s looking good. Notice that to handle a test of something which should throw an exception, you can either wrapper the function in a try/catch loop and, in the try side of the loop, have $this->assertTrue(false) to prevent false positives and in the catch side, do your $this->assertBlah() on the exception. Alternatively, (and much more simplely), use a documentation notation of @expectedException NameOfException and then prefix the function you are testing with the @ symbol. This is how I did it with the test “testFailLoadingConfig()”.

This obviously doesn’t handle setting and getting configuration values, so let’s add those.

Here’s the additions to the Config.php file:

    public function set($key = null, $value = null)
        if ($key == null) {
            throw new BadFunctionCallException("Key not set");
        if ($value == null) {
            unset ($this->arrValues[$key]);
            return true;
        } else {
            $this->arrValues[$key] = $value;
            return true;

    public function get($key = null)
        if ($key == null) {
            throw new BadFunctionCallException("Key not set");
        if (isset($this->arrValues[$key])) {
            return $this->arrValues[$key];
        } else {
            return null;

And the default.config.php file:

$this->set('demo', true);

And lastly, the changes to the ConfigTest.php file:

    public function testLoadConfig()
        $config = new Config();

     * @expectedException BadFunctionCallException
    public function testFailSettingValue()
        $config = new Config();

     * @expectedException BadFunctionCallException
    public function testFailGettingValue()
        $config = new Config();

We’ve not actually finished testing this yet. Not sure how I can tell?

phpunit --coverage-text Tests/ConfigTest.php
PHPUnit 3.6.10 by Sebastian Bergmann.


Time: 0 seconds, Memory: 3.75Mb

OK (4 tests, 5 assertions)

Generating textual code coverage report, this may take a moment.

Code Coverage Report
  2012-05-08 18:54:16

  Classes: 0.00% (0/1)
  Methods: 0.00% (0/3)
  Lines:   76.19% (16/21)

  Methods: 100.00% ( 3/ 3)   Lines:  76.19% ( 16/ 21)

Notice that there are 5 lines outstanding – probably around the unsetting values and using default values. If you use an IDE (like NetBeans) you can actually get the editor to show you, using coloured lines, exactly which lines you’ve not yet tested! Nice.

So, the last thing to talk about is Containers and Dependency Injection. We’ve already started with the Dependency Injection here – that $config->load(‘filename’); function handles loading config files, or you could just bypass that with $config->set(‘key’, ‘value); but once you get past a file or two, you might just end up with a lot of redundant re-loading of config files, or worse, lots of database connections open.

So, this is where Containers come in (something I horrifically failed to understand before).

Here’s a container:

class ConfigContainer
  protected static $config = null;

  public static function Load()
    if (self::$config == null) {
      self::$config = new Config();
    return self::$Config;

It’s purpose (in this case) is to load the config class, including any dependencies that you may need for that class, and then return that class to you. You could conceivably create a Database container, or a Request container or a User container with very little extra work, and with a few short calls, have a single function for each of your regular and routine sources of processing data, but without preventing you from being able to easily and repeatably test that data – by not going through the container.

Of course, there’s nothing to stop you just having these created in a registry class, or store them in a global from the get-go, but, I am calling these “Better Practices” after all, and these are considered to be not-so-good-practices.

Just as a note, code from this section can be seen at GitHub, if you want to use them at all.

Update 2012-05-11: Added detail to the try/catch exception catching as per frimkron’s comment. Thanks!

Logitech Media Server vs Ubuntu 12.04

A while back I upgraded my home server to Ubuntu 12.04 (while it was still in beta) and immediately the first thing I noticed was that the Logitech Media Server (previously known as Squeezebox Server) had stopped working.

Checking through the logs, I saw a lot of messages about perl dependencies being missing or not working [1]. As I was a bit busy at the time (the decision to upgrade had been due to something else entirely), I put it to one side (much to my wife’s annoyance!) to pick up later.

As I’ve been wallowing at home the past couple of days with a stomach bug, and not really been fit to do much other than moan, lie there and feel sorry about myself, I thought about what I could do to get my squeezebox server back up and running.

A few Google searches later, and I turn up this page: which suggests that this message below is due to Ampache… which now that I look at the log entry, it kinda makes sense. Queue digging into the depths of the server.

Under Ubuntu, all the serious configuration for the server is stored in /var/lib/squeezeboxserver, which includes the plugins.

So, firstly, I deleted the downloaded Zip file from /var/lib/squeezeboxserver/cache/DownloadedPlugins/

rm /var/lib/squeezeboxserver/cache/DownloadedPlugins/

Next, I removed the unpacked plugin from /var/lib/squeezeboxserver/cache/InstalledPlugins/Ampache

rm -Rf /var/lib/squeezeboxserver/cache/InstalledPlugins/Ampache

I made a mistake here on my box, and restarted the server. Woohoo it came back up, but the first thing it did was to re-download the plugin again! D’oh. So, now I need to find what’s telling it to re-install the plugin. Queue a quick grep. Ahhh, there’s a file called extensions.prefs, which says:

prefs/plugin/extensions.prefs:  Ampache: 1

And another file called state.prefs which says:

prefs/plugin/state.prefs:Ampache: needs-install

Note, these are both the output from grep – so the filename includes the path from the point /var/lib/squeezeboxserver. A quick nano away (or whatever editor you prefer) and I’d removed the line from the extensions.prefs which showed Ampache: 1, but the state.prefs was marginally more tricky. In here it lists them in three states, enabled, disabled and needs-install. So, I changed it to show disabled and then restarted the service. Tada. I’ve got a working Logitech Media Server again! Huzzah!

[1] Log file looks like this:

Slim::bootstrap::tryModuleLoad (285) Warning: Module [Plugins::GrabPlaylist::Plugin] failed to load:
Can't locate Math/ in @INC (@INC contains: /usr/sbin/Plugins/Gallery /var/lib/squeezeboxserver/cache/InstalledPlugins/Plugins/Gallery /var/lib/squeezeboxserver/cache/Installed
Plugins/Plugins/CustomScan/lib /var/lib/squeezeboxserver/cache/InstalledPlugins/Plugins/Ampache/lib CODE(0xb911c40) /var/lib/squeezeboxserver/cache/InstalledPlugins /usr/share/squeezebo
xserver/CPAN/arch/5.14/i386-linux-thread-multi-64int /usr/share/squeezeboxserver/CPAN/arch/5.14/i386-linux-thread-multi-64int/auto /usr/share/squeezeboxserver/CPAN/arch/5.14.2/i686-linu
x-gnu-thread-multi-64int /usr/share/squeezeboxserver/CPAN/arch/5.14.2/i686-linux-gnu-thread-multi-64int/auto /usr/share/squeezeboxserver/CPAN/arch/5.14/i686-linux-gnu-thread-multi-64int
 /usr/share/squeezeboxserver/CPAN/arch/5.14/i686-linux-gnu-thread-multi-64int/auto /usr/share/squeezeboxserver/CPAN/arch/i686-linux-gnu-thread-multi-64int /usr/share/squeezeboxserver/li
b /usr/share/squeezeboxserver/CPAN /usr/share/squeezeboxserver /usr/sbin /etc/perl /usr/local/lib/perl/5.14.2 /usr/local/share/perl/5.14.2 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/
5.14 /usr/share/perl/5.14 /usr/local/lib/site_perl . CODE(0xb911e20)) at Slim/Player/ line 18.
BEGIN failed--compilation aborted at Slim/Player/ line 18.
Compilation failed in require at /var/lib/squeezeboxserver/cache/InstalledPlugins/Plugins/GrabPlaylist/ line 22.
BEGIN failed--compilation aborted at /var/lib/squeezeboxserver/cache/InstalledPlugins/Plugins/GrabPlaylist/ line 22.
Compilation failed in require at (eval 924) line 2.
BEGIN failed--compilation aborted at (eval 924) line 2.

Nice, right?

Dependency Hell – and it’s not my package manager to blame

I’m writing a web service for conferences. I’ve been writing it, on and off for 3 years, and I think it would be fair to say the coding reflects my learning over those three years. The script is written in PHP. It has a MySQL backend. It has undergone a lot of changes.

In the beginning, was a concept. The concept was to have a digital timetable. One where you could register your interest in a talk, and the busiest talks got the biggest rooms. It should use any tech the user had to use the system including SMS. It would not expect you to have a user name and password combo (heaven forbid!), but would use OpenID.

The concept was implemented and demo’d at an event 2 years ago, and a friend [1] asked where the API was. “API?” I replied, feeling somewhat foolish, “It doesn’t have an API“.

[1] Lorna Jane Bowman (neé Mitchell aka @lornajane)

I realised the foolishness of my coding when discussing this issue [2]. “It’s dead simple, just create another view which returns just JSON” said someone. “View? I don’t know what you mean“… “Model, View, Controller? MVC – it’s a pretty common pattern”. “Oh no” I replied, “my project has inline PHP. It just seemed simpler that way.” “Well, how about you add a toString() function to your classes, and use that to render the output?” “Classes. Another thing my project doesn’t have. Sorry. Can’t do that either.

[2] With Lorna again, and adding Katherine Reeve (aka @BinaryKitten)

Did you ever get that slightly sinking feeling that maybe you’re making a fool of yourself?

“Well, there are lots of benefits to Classes in PHP, not least of which that you can use PHP CodeSniffer to enforce your coding standards” I start to laugh a little at this point “and you can use PHP Documenter to create all your developer documentation” which shut me right back up again “you can use PDO to create your objects from your database results” wow… mind blown – I’ve jumped on this little trick of a pony… “and of course, you can’t really do unit testing without classes”. Uh oh. “What’s unit testing?” “It’s a set of automated tests you can run against your code every time you commit to your version control software” whew! I’m using that at least! “to make sure that the code you’re committing hasn’t broken anything”.

Fast forward to this week, and I asked on Facebook whether anyone could teach me how to do Unit Testing. See, I’ve managed to cobble together my own MVC – I have classes called “Object_” which are the Models, the controller is my routing script – otherwise known as index.php, and then the Views are templates in Smarty, or just a json_encode((array) $object) [3] for my API. I have my own set of tests – not unit tests, it’s a script called “” which runs PHP against the file (to make sure that the pages don’t have brackets missing or similar), then runs PHP Code Sniffer against it, and finally, if all the files are OK, it then runs PHPDoc against the whole mass of it.

[3] As the Apple iPhone/iPad adverts say – some sequences shortened

So, one of the books suggested to me, again by the lovely Lorna Jane, was The Grumpy Programmer’s Guide To Building Testable PHP Applications which mentioned that it’s much easier to do unit testing if you set up dependency injection. Note, I’m still not yet doing Unit testing here.

Woah. What the hell is Dependency Injection? Well, fortunately, there were code examples in that book. Oh boy, were there code examples. So let’s look through this idea. Instead of writing

$stuff = new MyStuff();

class MyStuff()
    function construct()
        $this->db = mysql_connect('localhost', 'username', 'password');
        mysql_use_database('production', $this->db);

You could instead write this:

$db = mysql_connect('localhost', 'username', 'password');
mysql_use_database('production', $this->db);
$stuff = new MyStuff($db);

class MyStuff()
    function construct($db = null)
        $this->db = $db;

So, this now means that in your testing framework, you could pass it a non-production database, or a testing database, or, well, anything that relies on something outside your script.

I did a bit of digging around to find some other examples of dependency injection code that might be a bit easier to use, which is to say, something so I don’t need to amend all my constructor functions.

I found this slideshare from a talk at PHP Barcelona about dependency injection which says you can do dependency injection like this:

$thing = new MyClass($dependency);


$thing = new MyClass();


$thing = new MyClass();
 $thing->dependency = $dependency;

but somewhat weirdly, it also says that you can create a container class which holds your dependencies, and refer to that later – and that this isn’t a singleton. Sadly, I didn’t understand all that code fully (and have gone on to file a bug for the PHP documentation for the functions I didn’t understand to help people who follow behind with it!), but (and I’ve copied this verbatim from the slideshare) essentially, it looks like this:

class Container { protected $values = array(); function __set($id,$value) {  $this->values[$id] = $value;  }  function __get($id) {  if (!isset($this->values[$id])) {  throw new InvalidArgumentException(sprintf('Value "%s" is not defined.', $id));  }  if (is_callable($this->values[$id])) {  return $this->values[$id]($this);  } else { return $this->values[$id];  }  }  function asShared ($callable) {  return function($c) use ($callable) {  static $object; if (is_null($object)) {  $object=$callable($c);  }  return $object; }; } } $container = new Container(); $container->session_name='SESSION_ID'; $container->storage_class='SessionStorage'; $container->user = $container->asShared( function($c) {  return new User($c->storage);  } ); $container->storage = $container->asShared(  function($c) {  return new $c->storage_class($c->session_name); } );

Now, I have to be honest, this confuses the hell out of me. How on earth do I use this in my code? I’ve been doing this in my code thus far:

class Object_User{
  protected $intUserID    = null; // Obtained by the getCurrentUser() function
  protected $strUsername  = null;
  protected $hashPassword = null; // More of these, depending on the SQL

  function getCurrentUser() { // Called as $user = Base_User::getCurrentUser();
 $objCache  = Base_Cache::getHandler(); // A singleton to "cache" any data we've pulled to save getting it repeatedly
    if (
          && $objCache->arrCache['Object_User']['current'] != false
        ) {
      return $objCache->arrCache['Object_User']['current'];
    $arrRequest  = Base_Request::getRequest(); // Returns array of parsed request data
    $objDatabase = Base_Database::getConnection(); // Returns a PDO object

    $sql = "SELECT * FROM users WHERE strUsername = ? and hashPassword = ?";
    $query = $db->prepare($sql);
    $query->execute(array($request['username'], $request['password'])); $result = $query->fetchObject('Object_User');
    if ($result != false) {
      $objCache->arrCache['Object_User']['id'][$result->getVal('intUserID')] = $result;
      $objCache->arrCache['Object_User']['current'] = $result;
    return $result;

  function getVal($key) { // Return protected variables
    if (!isset($this->$key)) {
      return false;
    return $this->$key;

I know that singleton methods are considered “Bad” because they’re (apparently) difficult to unit test, but I would have thought that it would have been pretty straightforward to create a singleton class which holds all the dependencies (see following)

class Base_Dependencies
  protected static $handler = null; protected $arrDependencies = array();   protected function GetHandler() {
    if (self::$handler == null) { self::$handler = new self(); }
    return self::$handler;

  function set($key, $dependency) {
    $handler = self::GetHandler();
    $handler->arrDependencies[$key] = $dependency;

  function get($key) {
    $handler = self::GetHandler();
    if (isset($handler->arrDependencies[$key])) {
      return $handler->arrDependencies[$key];
    } else {
      return false;

  function unset($key) { // Only used for Unit Testing, I would imagine
    $handler = self::GetHandler();
    if (isset($handler->arrDependencies[$key])) {

Doing it this way means I can, from, for example, my database class, which is currently a singleton, say instead:

function GetConnection() {
  $db = Base_Dependencies::get("Database");
  if ($db != false) {
    return $db;
  $config = Base_Config::getAllConfig();
  $db = new PDO($config['DSN'], $config['DB_User'], $config['DB_Pass']);
  Base_Dependencies::set("Database", $db);
  return $db;

Is this wrong? Is this just not best practice? Given the above, how can I fix my dependencies in such a way that the poor schmuck who wants to commit a patch can figure out what they’re sending? Or do I just need to fix how my head works with this stuff? If it’s the latter, can someone provide some samples of what I’m doing wrong?

Thanks for reading this mammoth post!

Installing MOTP-AS under Ubuntu 11.10

Please note, I am having issues with localhost authentication. See below

MOTP-AS is a simple installable two-factor authentication system using the mOTP algorythm for generating one-time passwords. MOTP-AS integrates with FreeRadius to provide the same authentication to log in to managed servers in a consistent manner.

I’ve recently installed this on my Ubuntu 11.10 laptop and on my Ubuntu 12.04 Beta server, and the installation instructions worked on both, so I thought I’d share them with you.

Installing appropriate packages

sudo apt-get install libpam-radius-auth freeradius mysql-server phpmyadmin

Alternatively, use tasksel to install the LAMP server task, then

sudo apt-get install libpam-radius-auth freeradius

Download the latest version of motp-as from

Unpack it.

tar xfz ~/Downloads/motp-as*

Setting up the database

Go into the Setup/MySQL directory of the MOTP-AS directory. Edit motp_schema.sql at the line “CREATE USER”. Change the password from motp to something more secure.

mysql -u root -p < motp_schema.sql

Now update Setup/config.php with the new password you just created.

Setting up the web site

Copy the HTML directory to /var/www/motp (or somewhere else in your web root). You may need to do this either as root, or as a user with permissions to write to /var/www

cp -Rf ~/MOTP-AS_*/HTML /var/www/motp

Note this must be done after you’ve made your changes to Setup/config.php

Setting up FreeRadius

Stop the FreeRadius service

sudo /etc/init.d/freeradius stop


Backup the users file

sudo mv /etc/freeradius/users /etc/freeradius/users.dist

Edit the users file you’re about to copy in

nano ~/MOTP-AS_*/Setup/Freeradius/users

Find the part where it says “/var/www/htdocs/radius-auth.php” and change that to “/var/www/motp/radius-auth.php

Copy in the new users file

sudo cp ~/MOTP-AS_*/Setup/Freeradius/users /etc/freeradius/users

Dynamic Clients

Backup the dynamic-clients file

sudo mv /etc/freeradius/sites-available/dynamic-clients /etc/freeradius/sites-available/dynamic-clients.dist

Edit the new dynamic-clients file

nano ~/MOTP-AS_*/Setup/Freeradius/dynamic-clients

Find the three lines saying “/var/www/htdocs” and replace that string with “/var/www/motp” (I use Ctrl+W, Ctrl+R in nano to do a replace-all.)

Copy in the new dynamic-clients file

sudo cp ~/MOTP-AS_*/Setup/Freeradius/dynamic-clients /etc/freeradius/sites-available/dynamic-clients

Then make that function available

sudo ln -s /etc/freeradius/sites-available/dynamic-clients /etc/freeradius/sites-enabled/dynamic-clients


Amend the default script to enable accounting

sudo cp /etc/freeradius/sites-available/default /etc/freeradius/sites-available/default.dist

Then edit it to use the MOTP accounting functions

sudo nano /etc/freeradius/sites-available/default

Search for the line “accounting {” then comment that whole block out with the hash/pound sign “#“. Fortunately in the distribution supplied default file, this only means commenting out a few lines, which are “detail“, “unix“, “radutmp“, “exec“, “attr_filter.accounting_response“, and then the closing “}” for that block.

If you’re using nano, press the insert key (or Ctrl+R if you can’t find that easily) and enter /home/MyUserName/MOTP-AS_v0.7.2/Setup/Freeradius/accounting (amend the path as appropriate). Replace the section “/var/www/htdocs” with “/var/www/motp“.

Save and exit

Finishing off FreeRadius

sudo /etc/init.d/freeradius start

Install your client

Personally, I have an Android device, and I chose to install the Mobile-OTP app from the Android Marketplace. I also, through work, have a Nokia 6303i Classic, on which I installed the MOTP application from the MOTP site.

I’ve heard good things about iOTP for iPhone, although I personally don’t have one.

Configuring MOTP

Go to http://localhost/motp (or

Login with the username admin and password of motp.

Securing the admin account

Click on the red text in “First time configuration

Click on “Change password of User ‘admin’

Enter a new password. Do not set the time or uses section of this page. Click “Set“. Ignore the warning.

Click on “Home

Setting up your first user

Click on “Quick Add” (under “Wizards”)

Enter a username. It should be the username for your Ubuntu 11.10 device.

On the client, create a profile for the device. Most of them create a profile by asking for a seed, rather than a secret, so those will likely be more than 16 characters long – maybe even 20 (Mobile-OTP for Android) or 25 (MOTP Java app).

Once you’ve got your secret (on Mobile-OTP, by pushing-and-holding on the profile name and selecting “Show Secret“, on MOTP Java app, once you’ve put 0000 as the PIN for the first time to initialize it, you get a string “Init-Secret:“), put that into the “Secret” field, and then ask the user to set their pin here – I suggest 1234 initially, as the user can change it to something they want after.

Click OK, then click “Logout” and test authentication. If it all goes OK, they should be presented with “Welcome to the Mobile OTP Authentication Server“.

Under “Settings” they can change their own PIN.

Testing radius authentication works OK

Run the radius testing program, like this, as a user:

radtest username passcode localhost 0 testing123

(This assumes the default localhost password hasn’t changed)

If you get anything like “rad_recv: Access-Reject packet from host“, then you’ve failed to configure something properly, or you’ve entered the PIN or code wrong.

Restart FreeRadius in debugging mode by doing the following:

/etc/init.d/freeradius stop
/usr/sbin/freeradius -X

This will produce a large quantity of logs on-screen, so I’d suggest running the test itself from a separate window. Run the radtest command (listed above) again. Look for your error messages. In my case, I forgot to update the line in users, so I saw this error message: Could not open input file: /var/www/htdocs/radius-auth.php

To find where this fault was, I did (as root, in /etc/freeradius)

find -R 'htdocs' /etc/freeradius

And got back: users: Exec-Program-Wait = “/usr/bin/php /var/www/htdocs/radius-auth.php %{User-Name} %{User-Password} %{Client-Shortname}”

That told me the fault was in the users file.

Fix the issue, check it again, and when you get this message “rad_recv: Access-Accept packet from host” press Ctrl+C to cancel the test mode of FreeRadius, and then run:

sudo /etc/init.d/freeradius start

Configuring pam_radius_auth.conf

Edit /etc/pam_radius_auth.conf

sudo nano /etc/pam_radius_auth.conf

Find the line which says “” and replace the shared secret with something you want your server to use. You will also need to amend /etc/freeradius/clients.conf and replace the “secret” in the localhost client there (by default, it’s “testing123” in freeradius).

If you want to use your OTP for all authentication credentials, edit /etc/pam.d/common-auth, or if you just want to use it with specific access protocols, edit the relevant file in /etc/pam.d for the authentication systems you want to use OTP for.

You need to add the following line – either on the line before “@include common-auth” (for non common-auth files) or after the primary comment block for common-auth.

auth sufficient

Open a separate terminal session to your box (especially! if you’re remote) and ensure you can still login with your regular credentials.

Then try a connection with your radius credentials. It should just work! If not, stop the freeradius server and re-run it using /usr/sbin/freeradius -X and see whether you’re getting a different error message.

** UPDATE **

I have noticed that I’m getting locked out when using my non-radius credentials. This is probably due to the placement of the line in the /etc/pam.d/common-auth – it should probably come after the line, but I’ve not tested that yet. I’m also going to try to suggest that there be an optional time-out period on locked accounts to the developers of MOTP-AS.

The second issue I’m struggling with is that I’m getting errors when using the LightDM. I’m getting the following error message in /var/log/auth.log:

pam_succeed_if(lightdm:auth): requirement "user ingroup nopasswdlogin" not met by user "spriggsj"

I don’t know if this is because I’m using ecryptfs as well, or because there’s something wonky going on with the common-auth structure I’m using.