Tag: Projects

One to read: “Test Driven Development (TDD) for networks, using Ansible”

JonTheNiceGuy Leave a comment

Thanks to my colleague Simon (@sipart on Twitter), I spotted this post (and it’s companion Github Repository) which explains how to do test-driven development in Ansible.

Essentially, you create two roles – test (the author referred to it as “validate”) and one to actually do the thing you want it to do (in the author’s case “add_vlan”).

In the testing role, you’d have the following layout:

/path/to/roles/testing/tasks/main.yml
/path/to/roles/testing/tasks/SOMEFEATUREtest.yml

In the main.yml file, you have a simple stanza:

---
- name: Include all the test files
  include: "{{ outer_item }}"
  with_fileglob:"/path/to/roles/validate/tasks/*test.yml"
  loop_control: loop_var=outer_item

I’m sure that “with_fileglob” line could be improved to not actually need a full path… anyway

Then in your YourFeature_test.yml file, you do things like this:

---
- name: "Pseudocode in here. Use real modules for your testing!!"
  get_vlan_config: filter_for=needle_vlan
  register:haystack_var

- assert: that=" {{ needle_item }} in haystack_var "

When you run the play of the role the first time, the response will be “failed” (because “needle_vlan” doesn’t exist). Next do the “real” play of the role (so, in the author’s case, add_vlan) which creates the vlan. Then re-run the test role, your response should now be “ok”.

I’d probably script this so that it goes:

      reset-environment set_testing=true (maybe create a random little network)
      test
      run-action
      test
      reset-environment set_testing=false

The benefit to doing it that way is that you “know” your tests aren’t running if the environment doesn’t have the “set_testing” thing in place, you get to run all your tests in a “clean room”, and then you clear it back down again afterwards, leaving it clear for the next pass of your automated testing suite.

Fun!

Today I learned… Ansible Include Templates

JonTheNiceGuy 1 Comment

I am building Openstack Servers with the ansible os_server module. One of these fields will accept a very long string (userdata). Typically, I end up with a giant blob of unreadable build script in this field…

Today I learned that I can use this:

---
- name: "Create Server"
  os_server:
    name: "{{ item.value.name }}"
    state: present
    availability_zone: "{{ item.value.az.name }}"
    flavor: "{{ item.value.flavor }}"
    key_name: "{{ item.value.az.keypair }}"
    nics: "[{%- for nw in item.value.ports -%}{'port-name': '{{ ProjectPrefix }}{{ item.value.name }}-Port-{{nw.network.name}}'}{%- if not loop.last -%}, {%- endif -%} {%- endfor -%}]" # Ignore this line - it's complicated for a reason
    boot_volume: "{{ ProjectPrefix }}{{ item.value.name }}-OS-Volume" # Ignore this line also :)
    terminate_volume: yes
    volumes: "{%- if item.value.log_size is defined -%}[{{ ProjectPrefix }}{{ item.value.name }}-Log-Volume]{%- else -%}{{ omit }}{%- endif -%}"
    userdata: "{% include 'templates/userdata.j2' %}"
    auto_ip: no
    timeout: 65535
    cloud: "{{ cloud }}"
  with_dict: "{{ Servers }}"

This file (/path/to/ansible/playbooks/servers.yml) is referenced by my play.yml (/path/to/ansible/play.yml) via an include, so the template reference there is in my templates directory (/path/to/ansible/templates/userdata.j2).

That template can also then reference other template files itself (using {% include 'templates/some_other_file.extension' %}) so you can have nicely complex userdata fields with loads and loads of detail, and not make the actual play complicated (or at least, no more than it already needs to be!)

Some notes from Ansible mentoring

JonTheNiceGuy Leave a comment

Last night, I met up with my friend Tim Dobson to talk about Ansible. I’m not an expert, but I’ve done a lot of Ansible recently, and he wanted some pointers.

He already had some general knowledge, but wanted some pointers on “other things you can do with Ansible”, so here are a couple of the things we did.

  • If you want to set certain things to happen as “Production” and other things to happen as “Pre-production” you can either have two playbooks (e.g. pre-prod.yml versus prod.yml) which call certain features… OR use something like this:

    ---
    - hosts: localhost
      tasks:
    - set_fact:
        my_run_state: "{% if lookup('env', 'runstate') == '' %}{{ default_run_state|default('prod') }}{% else %}{{ lookup('env', 'runstate')|lower() }}{% endif %}"
    - debug: msg="Doing prod"
      when: my_run_state == 'prod'
    - debug: msg="Doing something else"
      when: my_run_state != 'prod'

    With this, you can define a default run state (prod), override it with a group or host var (if you have, for example, a staging service or proof of concept space), or use your Environment variables to do things. In the last case, you’d execute this as follows:

    runstate=preprod ansible-playbook site.yml

  • You can tag almost every action in your plays. Here are some (contrived) examples:


    ---
    - name: Get facts from your hosts
      tags: configure
      hosts: all
    - name: Tell me all the variable data you've collected
      tags: dump
      hosts: localhost
      tasks:
        - name: Show data
          tags: show
          debug:
            var=item
          with_items: hostvars

    When you then run

    ansible-playbook test.yml --list-tags

    You get

    playbook: test.yml

      play #1 (all): Get facts from your hosts      TAGS: [configure]
          TASK TAGS: [configure]

      play #2 (localhost): Tell me all the variable data you've collected   TAGS: [dump]
          TASK TAGS: [dump, show]

    Now you can run ansible-playbook test.yml -t configure or ansible-playbook test.yml --skip-tags configure

    To show how useful this can be, here’s the output from the “–list-tags” I’ve got on a project I’m doing at work:
    playbook: site.yml

      play #1 (localhost): Provision A-Side Infrastructure  TAGS: [Functional_Testing,A_End]
          TASK TAGS: [A_End, EXCLUDE_K5_FirewallManagers, EXCLUDE_K5_Firewalls, EXCLUDE_K5_Networks, EXCLUDE_K5_SecurityGroups, EXCLUDE_K5_Servers, Functional_Testing, K5_Auth, K5_FirewallManagers, K5_Firewalls, K5_InterProjectLinks, K5_Networks, K5_SecurityGroups, K5_Servers]

      play #2 (localhost): Provision B-Side Infrastructure  TAGS: [Functional_Testing,B_End]
          TASK TAGS: [B_End, EXCLUDE_K5_Firewalls, EXCLUDE_K5_Networks, EXCLUDE_K5_SecurityGroups, EXCLUDE_K5_Servers, Functional_Testing, K5_Auth, K5_FirewallManagers, K5_Firewalls, K5_InterProjectLinks, K5_Networks, K5_SecurityGroups, K5_Servers]

      play #3 (localhost): Provision InterProject Links - Part 1    TAGS: [Functional_Testing,InterProjectLink]
          TASK TAGS: [EXCLUDE_K5_InterProjectLinks, Functional_Testing, InterProjectLink, K5_InterProjectLinks]

      play #4 (localhost): Provision InterProject Links - Part 2    TAGS: [Functional_Testing,InterProjectLink]
          TASK TAGS: [EXCLUDE_K5_InterProjectLinks, Functional_Testing, InterProjectLink, K5_InterProjectLinks]

      play #5 (localhost): Provision TPT environment        TAGS: [Performance_Testing]
          TASK TAGS: [EXCLUDE_K5_FirewallManagers, EXCLUDE_K5_Firewalls, EXCLUDE_K5_Networks, EXCLUDE_K5_SecurityGroups, EXCLUDE_K5_Servers, K5_Auth, K5_FirewallManagers, K5_Firewalls, K5_InterProjectLinks, K5_Networks, K5_SecurityGroups, K5_Servers, Performance_Testing, debug]

    This then means that if I get a build fail part-way through, or if I’m debugging just a particular part, I can run this: ansible-playbook site.yml -t Performance_Testing --skip-tags EXCLUDE_K5_Firewalls,EXCLUDE_K5_SecurityGroups,EXCLUDE_K5_Networks

Using Python-OpenstackClient and Ansible with K5

JonTheNiceGuy Leave a comment

Recently, I have used K5, which is an instance of OpenStack, run by Fujitsu (my employer). To do some of the automation tasks I have played with both python-openstackclient and Ansible. This post is going to cover how to get those tools to work with K5.

I have access to a Linux virtual machine (Ubuntu 16.04) and the Windows Subsystem for Linux in Windows 10 to run “Bash on Ubuntu on Windows”, and both accept the same set of commands.

In order to run these commands, you need a couple of dependencies. Your mileage might vary with other Linux distributions, but, for Ubuntu based distributions, run this command:

sudo apt install python-pip build-essential libssl-dev libffi-dev python-dev

Next, use pip to install the python modules you need:

sudo -H pip install shade==1.11.1 ansible cryptography python-openstackclient

If you’re only ever going to be working with a single project, you can define a handful of environment variables prefixed OS_, like this:

export OS_USERNAME=BloggsF
export OS_PASSWORD=MySuperSecretPasswordIsHere
export OS_REGION_NAME=uk-1
export OS_USER_DOMAIN_NAME=YourProjectName
export OS_PROJECT_NAME=YourProjectName-prj
export OS_PROJECT_ID=baddecafbaddecafbaddecafbaddecaf
export OS_AUTH_URL=https://identity.uk-1.cloud.global.fujitsu.com/v3
export OS_VOLUME_API_VERSION=2
export OS_IDENTITY_API_VERSION=3

But, if you’re working with a few projects, it’s probably worth separating these out into clouds.yml files. This would be stored in ~/.config/openstack/clouds.yml with the credentials for the environment you’re using:

---
clouds:
  root:
    identity_api_version: 3
    regions:
    - uk-1
    auth:
      auth_url: https://identity.uk-1.cloud.global.fujitsu.com/v3
      password: MySuperSecretPasswordIsHere
      project_id: baddecafbaddecafbaddecafbaddecaf
      project_name: YourProjectName-prj
      username: BloggsF
      user_domain_name: YourProjectName

Optionally, you can separate out the password, username or any other “sensitive” information into a secure.yml file stored in the same location (removing those lines from the clouds.yml file), like this:

---
clouds:
  root:
    auth:
      password: MySuperSecretPasswordIsHere

Now, you can use the Python based Openstack Client, using this invocation:

openstack --os-cloud root server list

Alternatively you can use the Ansible Openstack (and K5) modules, like this:

---
tasks:
- name: "Authenticate to K5"
  k5_auth:
    cloud: root
  register: k5_auth_reg
- name: "Create Network"
  k5_create_network:
    name: "Public"
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Create Subnet"
  k5_create_subnet:
    name: "Public"
    network_name: "Public"
    cidr: "192.0.2.0/24"
    gateway_ip: "192.0.2.1"
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Create Router"
  k5_create_router:
    name: "Public"
    availability_zone: "uk-1a"
    state: present
    k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Attach private network to router"
  os_router:
    name: "Public"
    state: present
    network: "inf_az1_ext-net02"
    interfaces: "Public"
    cloud: root
- name: "Create Servers"
  os_server:
    name: "Server"
    availability_zone: "uk-1a"
    flavor: "P-1"
    state: present
    key_name: "MyFirstKey"
    network: "Public-Network"
    image: "Ubuntu Server 14.04 LTS (English) 02"
    boot_from_volume: yes
    terminate_volume: yes
    security_groups: "Default"
    auto_ip: no
    timeout: 7200
    cloud: root

Using riot.im as a bridge to Freenode

JonTheNiceGuy Leave a comment

Over the last few months, I’ve been using Riot.im (the client for Matrix.org) as my primary IRC client, and access to other end-to-end-encrypted chats.

A few weeks ago I decided I wanted to use my “normal” IRC nickname on Freenode, so looked into how to do it. It’s surprisingly easy, but there are a few gotchas.

Making Matrix know your password

First of all you need to message the IRC bridge bot and tell it your nickserv password: !storepass chat.freenode.net MyComplexPassword

Next, you need to chat with NickServ directly and authenticate with it: identify MyNormalNick MyComplexPassword

Lastly, you go back to your chat with the bridge bot, and tell it your nickname: !nick chat.freenode.net MyNormalNick

If, in the process of doing this, you find you can’t log in as yourself, message NickServ and tell it to release your account from being protected: release MyNormalNick MyComplexPassword

Setting topics

While the bridge bot will let you set a topic (!cmd TOPIC #channel Something) this didn’t really work for me, so instead, I use ChanServ to do it for me: topic #channel My New Topic

Just remember that you must have ops granted to you for that channel through ChanServ to be able to make such changes.

You can also set modes for people, or the channel, ban people, voice or op them through ChanServ, just send the command help in the chat to ChanServ for more guidance!

And, if you’re stuck, feel free to come ask for help! I’m (predictably) JonTheNiceGuy!

Building a Dual boot machine running Ubuntu 17.04 and Windows 10 with full-disk encryption

JonTheNiceGuy 4 Comments

This post has been revised since it was initially published on 31st March due to errors found in the resulting build. It was also missing details on the shared data drive between the two machines, so has been amended to include that.

** WARNING ** This works for me – it might not for you!

The outcome of this build will leave you with the following:

Boot up, go through the VeraCrypt bootloader, enter a password for Windows, or press escape to load the Grub bootloader where you will boot (K|L|X|)Ubuntu(| Mate| Gnome).

The Windows environment will be encrypted with VeraCrypt, an open source Full Disk Encryption technology, while the Linux environment will be encrypted using Luks. The shared volume (between Windows and Linux) will be encrypted with VeraCrypt.

PLEASE BE AWARE THAT ANY WINDOWS 10 UPGRADES WILL FAIL TO APPLY AS IT WILL NOT RECOGNISE THE VERACRYPT FILE SYSTEM! To resolve this, decrypt the Windows volume, perform the upgrade, re-encrypt it, then transfer the new recovery ISO image to the boot volume, following the method below. Yes, this will take some time. No, you don’t need to decrypt the data volume. Yes, you can use that data volume to shunt the ISO image around.

LATE EDIT 2020-01-06: I’ve just spotted a link to this post over on Level1Techs. In that post, someone asked if the broken upgrades is still a thing. Turns out that since I wrote this in 2017, it’s not been fixed. Now, I should stress, I’ve stopped using this layout as I went all-Linux on that machine, but… it might work for you now?! Also, shout out to 92aceshigh for referencing this post, and glad something I wrote helped you! ☺

Step 1:Create your partition table

My partition table, for a 320GB Disk looks (roughly) like this:

Partition 1: 20GB – Linux /Boot (ext2, plus space for ISO files for random booting)
Partition 2: 60GB – Windows C:\ (NTFS VeraCrypt)
Partition 3: 72GB – Linux Physical Volume (LVM PV, Luks Encrypted)
– logical volume 1: 16Gb Swap (Linux Swap)
– logical volume 2: 60Gb Linux (ext4)
Partition 4: 156GB – Shared Volume (NTFS, VeraCrypt)

I performed this using GParted in the Gnome Live image using the GParted. Some rational here:

  1. The first partition also allows me to add other ISOs if I want to boot them.
  2. I have 4GB RAM, this gives me some extra space to allow me to hibernate, but also… 4Gb. Ugh.
  3. I then split my Linux and Windows partitions into two equal parts.

Step 2: Use Cryptsetup to format the disk

The following steps need to be run as root.

sudo -i

Step 2a: Format the partitions as LUKS

cryptsetup luksFormat -y -v /dev/sda3

Step 2b: Open the LUKS volume

cryptsetup luksOpen /dev/sda3 lvm-pv

Step 2c: Create the LVM Physical Volume over the LUKS volume

vgcreate vg00 /dev/mapper/lvm-pv

Step 2d: Define the LVM Logical Volumes over the LVM Physical Volume

lvcreate -n lv00_swap -L 16G vg00       # Define 16GB Swap Space
lvcreate -n lv01_root -l +100%FREE vg00 # Define the rest of vg00 as /

LEAVE YOUR TERMINAL OPEN

Step 3: Install your Linux distribution.

Note that when you perform your install, when you get to the partitioning screen, select “Manual”, and then pick out the following volumes:

/dev/mapper/vg00-lv01_root = ext4 formatted, mount point: /
/dev/mapper/vg00-lv00_swap = swap
/dev/sda1 = ext2, format, mount point: /boot

Select the boot volume of /dev/sda. But wait, I hear you say, Windows has a well know history of nuking Grub partitions… Well, we’ll sort that in a bit…

DON’T EXIT THE LIVE SESSION ONCE THE INSTALL HAS FINISHED (select “Continue Testing”).

Step 4: Make your machine actually able to boot

Go back to your terminal session.  It should still be logged in as root. We need to re-mount all the partitions…

Step 4a: Mount your volumes

mount /dev/mapper/vg00-lv01_root /target
mount /dev/sda1 /target/boot
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /target$i; done

Step 4b: Swap to the “Target” filesystem

chroot /target

Step 4c: Setup your volumes to prompt for cryptographic keys

echo "LinuxRoot UUID=`blkid | grep sda3 | cut -d\\\" -f2` none luks" > /etc/crypttab

Step 4d: Update the boot volume to use these changes

update-initramfs -u

Step 4e: Ensure Grub is also installed to the MBR for testing

grub-install --force /dev/sda1
chattr +i /boot/grub/i386-pc/core.img
update-grub

The first part installs grub to the boot position, even though it doesn’t like it, and the second forces the core file to be unchangeable… I’m not exactly sure of the impact of this, but it’s the only way to do the next part of this process. The last bit makes sure that you’ve got the latest grub config files installed.

Step 5: Reboot and test

Just check to make sure the machine boots OK!

You should have a booting Ubuntu derivative with an encrypted file system. Awesome.

Now let’s install Windows!

Step 6: Install Windows and Veracrypt.

You should boot from your install media, when you get to the partition selector, there should only be a single NTFS partition for it to use. Use it.

Install the latest version of Veracrypt from https://veracrypt.codeplex.com/

Once it’s installed, go to System, Encrypt System Partition/Drive, “Normal” system encryption, Windows System Partition, Multi-Boot (accept the warning), Boot Drive “Yes”, Single Disk, “Non-Windows Boot Loader” – No, and then… let it go through all the rest of the steps. There will be one stage where it asks you to create a rescue disk. Just save it for later. Once the encryption settings are collected, it will do a test (which is basically just rebooting to the boot loader, having you put in your password and going back into Windows), and then let it start performing the encryption.

Once the encryption finishes, reboot the computer, enter the decryption password and test it boots to Windows OK. Then reboot it again and press escape instead of putting in the password. It will boot to your Ubuntu system.

So, there you have it. One Dual-Boot system with encrypted disks everywhere!

Step 7: Setting up the shared volume.

After you’ve got the Ubuntu and Windows volumes sorted out, next we need the shared data volume to be organised. You’ll need Veracrypt for Ubuntu. Use the following to install the Veracrypt package for Ubuntu:

sudo add-apt-repository ppa:unit193/encryption
sudo apt-get update
sudo apt-get install veracrypt

Once that’s installed, boot back into Windows and create a new volume – perhaps V: for Veracrypt, or E: for Encrypted – your choice, but make sure you create it using the same password that you used for the Windows partition.

Format this new volume with either NTFS or FAT32 so that you can mount it under either operating system. I chose NTFS.

Now, you need to go into Veracrypt’s Settings menu, and select “System Encryption Settings”. Tick “Cache pre-boot authentication password in driver memory” (be aware, this means that if your machine is compromised when powered up, the password could be recovered), then OK. This may prompt you to accept the UAC at this point.

Next, with the mounted volume selected, go to the “Favorites” menu, and choose “Add to System Favorites”. In the screen which comes up, select the box under “Global Settings” which says “Mount system favorite volumes when Windows starts (in the initial phase of the startup procedure)”. There will be a warning about passwords that appears. Click OK.

You may, at this point, want to move certain aspects of your Windows desktop (e.g. the “My Documents” location) to the new mounted drive.

On the Linux OS, become root, with sudo, and then add the following lines to your crontab:

@reboot mkdir -p /shared_storage 
@reboot veracrypt --text --non-interactive --fs-options=uid=1000,gid=1000,umask=0077 --password="YOURSUPERSECUREPASSWORD" /dev/sda4 /shared_storage

These assume that your login user’s ID is 1000 (you can check that by running the command “id” as your logged in user), that you want to use “/shared_storage” as the mount point (it stops Ubuntu treating it as a “Mountable Volume” if it’s not in your home directory and not in /mnt or /media). These options also mean that only that user (and root) can access any of the files in that partition (although, it is only you on this laptop… right?), which means you can safely use it for any files which check user permissions before allowing you to access them (e.g. SSH keys). I then set up a symbolic link to /home/MYUSERACCOUNT/Documents into the /shared_storage/Documents directory, and /home/MYUSERACCOUNT/.ssh into the /shared_storage/SSH_Keys directory.

Citataions

The following list of resources helped me out when I was struggling with what to do next! They may not be canonical sources, but they helped.

  1. http://thesimplecomputer.info/full-disk-encryption-with-ubuntu – This is what got me started on this little journey!
  2. http://askubuntu.com/questions/161689/how-do-i-get-grub2-to-boot-a-truecrypt-encrypted-mbr – How to add the Veracrypt recovery disk to your Grub boot partition. Note, I do it slightly differently to this now.
  3. http://askubuntu.com/questions/711801/i-deleted-files-in-boot-now-cant-boot-linux – I may have done this. It tells you how to put all your important files back for booting purposes :)
  4. https://www.youtube.com/watch?v=Z1yWbBIqh1o – Walk through of installing Veracrypt to Windows 10. I used this to see some of the terms after I’d already installed Veracrypt. I don’t quite follow the same route as him though.
  5. https://www.linux.com/blog/how-full-encrypt-your-linux-system-lvm-luks – Using LVM inside Luks for the full-Linux disk (this was why I’ve re-written this post)
  6. https://wiki.archlinux.org/index.php/Dm-crypt/Encrypting_an_entire_system – Some details around how the Luks stuff all works

I may or may not have reinstalled Windows and Kubuntu about 20 times during this process, cursing myself for starting the whole damn process off in the first place!!!

Setting up a Google Play Music uploader for Linux Servers

JonTheNiceGuy 8 Comments

THIS POST HAS BEEN SUPERCEEDED.

I like having an online music server. At home, I use a Logitech Media Server (formerly Squeezebox Server) [1] and run my several O2 Jogglers around the house with the “Squeezeplay OS” [2] images to play music from that server, but when it comes to an Android tablet (or Android phone), there’s not that joined-up thinking from Logitech (although you can just about cobble it together using a few 3rd party apps [3]), but what I do like is the Google Play Music service.

Google Play Music [4] was the first product from the Google Play team after they rebranded to “Google Play”, and I pretty quickly got interested in it. I installed the Google Play Uploader [5] on my home server, and uploaded all my music (apparently, I’m up to 12,000 tracks, but I think there are some duplicates there!) but what to do about the rest of the family? Well, until just recently, it didn’t matter. Jules has no interest in playing music on her phone or tablet, and Daniel, well, he’s 3-and-a-bit.

Since pretty early on, he’s been all over our tech – initially just using whatever apps we had installed on Jules’ and my phones, then Jules’ tablet (I was, and still am, pretty cautious about him using my tablet as it’s an Asus TF300T [6] which cost quite a lot of money, and I keep toying with the idea of installing some other OS – like Firefox OS [7] or Ubuntu Touch [8] on there and can’t if it’s wrecked), and now he’s pretty comfortable with browsing YouTube or the Play Store, although he knows not to click on adverts and can’t install anything that costs anything.

In the last couple of months, since he’s been really learning how to spell, he’s been asking how to spell the names of his favourite films (“Oliver!”, “Frozen” and “The Polar Express” primarily) to get the film clips up in YouTube, or to play snippets from the soundtracks of the films… which got me thinking. I can’t really do much about the films (not yet at least!), but perhaps I could set up Google Play Music with the soundtracks he listens to… but I already have one account sync’ed with Google Play Music from my media server…. how do I get his stuff set up on there?

Essentially, Google Play Music Uploader is a GUI application that can’t be started in the CLI (which kinda makes sense from a simplicity perspective), but as I’m already running one instance of the application on my media server, I can’t start up a second one, so what do I do?

Well, as it turns out, there’s actually a python library for interacting with Google Play Music’s upload and download applications called “gmusicapi” [9], and this couples with a really nice wrapper gives me a CLI utility I can run in a CRON job on my media server.

The wrapper is called gmusicapi-scripts [10], which contains 3 utilities – gmdownload.py, gmsync.py and the key to this – gmupload.py.

You need to install a few libraries. On my Ubuntu 14.04 system, I needed to run:

sudo apt-get install python-pip avconv-tools
sudo pip install gmusicapi
sudo pip install docopt

Once you’ve got this, you can get the tools themselves like this:

git clone https://github.com/thebigmunch/gmusicapi-scripts.git

This will give you a folder called gmusicapi-scripts in which is gmupload.py – the first time you run it, it’ll ask you to visit a web page in order to register the client. Click “OK” to approve the library having access to Google Play Music. This is a pretty spartan page, and ends up with a grey text box containing a string. Copy the contents of that box back into your terminal, and hey-presto, you get it working…

Well, sort of. For me, because I’d not set Daniel up with Google Play Music yet, I needed to set up his account first. I didn’t realise this (I thought, just going to the web page the script points you to will get you access, but it doesn’t because they need to vet which country you’re accessing from…) and the script didn’t tell me that (it just kept saying “Not Subscribed” [11])

Anyway, once it’s done you run

~/gmusicapi-scripts/gmupload.py /path/to/file.mp3

( 1/1) Successfully uploaded /path/to/file.mp3

If you’ve got multiple users, you can rename ~/.local/share/gmusicapi/oauth.cred to ~/.local/share/gmusicapi/SomeOtherName.cred and then run

~/gmusicapi-scripts/gmupload.py -c SomeOtherName /path/to/file.mp3

Subsequent passes will prompt you to authenticate the next account as you go, and then you can rename them as appropriate.

[1] Logitech Media Server (formerly Squeezebox Server): http://www.mysqueezebox.com/download
[2] “Squeezeplay OS”: http://birdslikewires.co.uk/articles/squeezeplay-os
[3] a few 3rd party apps: I got it working on my TF300T by combining https://play.google.com/store/apps/details?id=de.bluegaspode.squeezeplayer and https://play.google.com/store/apps/details?id=de.cedata.android.squeezecommander
[4] Google Play Music: https://play.google.com/music
[5] Google Play Uploader: https://play.google.com/music/listen?u=0#/manager
[6] TF300T: http://en.wikipedia.org/wiki/Asus_Transformer_Pad_TF300T
[7] Firefox OS: https://www.mozilla.org/en-US/firefox/os/
[8] Ubuntu Touch: http://www.ubuntu.com/tablet
[9] “gmusicapi”: https://github.com/simon-weber/Unofficial-Google-Music-API
[10] gmusicapi-scripts: https://github.com/thebigmunch/gmusicapi-scripts
[11] “Not Subscribed”: I raised a bug, but the lead developer said it seemed obvious to him that you have to set it up first… not disputing that for the first pass, but a nice message would have been good :) https://github.com/thebigmunch/gmusicapi-scripts/issues/22

Starting EC2 instances using PHP

JonTheNiceGuy Leave a comment

I run a small podcast website called CCHits.net. It runs on Dreamhost because they offer unlimited storage and bandwidth, but while it’s a great service for storage, it’s not really useful for running a batch process because long running processes are killed regularly (in my case, building the cchits podcasts on a daily basis).

As a result, I built an EC2 instance which I trigger every day using a cronjob. Previously, I used the “AWS CLI tools”, but as this uses a Java Virtual Machine, it was taking an awful lot of resources just to spin up an instance, and Dreamhost kept killing the task off. As a result, I found the AWS PHP SDK, and coded up this little snippet to spin up the EC2 instance.

Development Environment Replication with Vagrant and Puppet

JonTheNiceGuy Leave a comment

This week, I was fortunate enough to meet up with the Cheadle Geeks group. I got talking to a couple of people about Vagrant and Puppet, and explaining how it works, and I thought the best thing to do would be to also write that down here, so that I can point anyone who missed any of what I was saying to it.

Essentially, Vagrant is program to read a config file which defines how to initialize a pre-built virtual machine. It has several virtual machine engines which it can invoke (see [1] for more details on that), but the default virtual machine to use is VirtualBox.

To actually find a virtual box to load, there’s a big list over at vagrantbox.es which have most standard cloud servers available to you. Personally I use the Ubuntu Precise 32bit image from VagrantUp.com for my open source projects (which means more developers can get involved). Once you’ve picked an image, use the following command to get it installed on your development machine (you only need to do this step once per box!):

vagrant box add {YourBoxName} {BoxURL}

After you’ve done that, you need to set up the Vagrant configuration file.

cd /path/to/your/dev/environment
mkdir Vagrant
cd Vagrant
vagrant init {YourBoxName}

This will create a file called Vagrantfile in /path/to/your/dev/environment/Vagrant. It looks overwhelming at first, but if you trim out some of the notes (and tweak one or two of the lines), you’ll end up with a file which looks a bit like this:

Vagrant.configure("2") do |config|
  config.vm.box = "{YourBoxName}"
  config.vm.hostname = "{fqdn.of.your.host}"
  config.vm.box_url = "{BoxURL}"
  config.vm.network :forwarded_port, guest: 80, host: 8080
  # config.vm.network :public_network
  config.vm.synced_folder "../web", "/var/www"
  config.vm.provision :puppet do |puppet|
    puppet.manifests_path = "manifests"
    puppet.manifest_file  = "site.pp"
  end
end

This assumes you’ve replaced anything with {}’s in it with a real value, and that you want to forward TCP/8080 on your machine to TCP/80 on that box (there are other work arounds, using more Vagrant plugins, different network types, or other services such as pagekite, but this will do for now).

Once you’ve got this file, you could start up your machine and get a bare box, but that’s not much use to you, as you’d have to tell people how to configure your development environment every time they started up a new box. Instead, we’ll be using a Provisioning service, and we’re going to use Puppet for that.

Puppet was originally designed as a way of defining configuration across all an estate’s servers, and a lot of tutorials I’ve found online explain how to use it for that, but when we’re setting up Puppet for a development environment, we just need a simple file. This is the site.pp manifest, and in here we define the extra files and packages we need, plus any commands we need to run. So, let’s start with a basic manifest file:

node default {

}

Wow, isn’t that easy? :) We need some more detail than that though. First, let’s make sure the timezone is set. I live in the UK, so my timezone is “Europe/London”. Let’s put that in. We also need to make sure that any commands we run have the right path in them. So here’s our revised, debian based, manifest file.

node default {
    Exec {
        path => '/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/sbin:/usr/sbin'
    }

    package { "tzdata":
        ensure => "installed"
    }

    file { "/etc/timezone":
        content => "Europe/London\n",
        require => Package["tzdata"]
    }

    exec { "Set Timezone":
        unless => "diff /etc/localtime /usr/share/zoneinfo/`cat /etc/timezone`",
        command => "dpkg-reconfigure -f noninteractive tzdata",
        require => File["/etc/timezone"]
    }
}

OK, so we’ve got some pretty clear examples of code to run here. The first Exec statement must always be in there, otherwise it gets a bit confused, but after that, we’re making sure the package tzdata is installed, we then make sure that, once the tzdata package is installed, we create or update the /etc/timezone file with the value we want, and then we use the dpkg-reconfigure command to set the timezone, but only if the timezone isn’t already set to that.

Just to be clear, this file describes what the system should look like at the end of it running, not a step-by-step guide to getting it running, so you might find that some of these packages install out of sequence, or something else might run before or after when you were expecting it to run. As a result, you should make good use of the “require” and “unless” statements if you want a proper sequence of events to occur.

Now, so far, all this does is set the timezone for us, it doesn’t set up anything like Apache or MySQL… perhaps you want to install something like WordPress here? Well, let’s see how we get other packages installed.

In the following lines of code, we’ll assume you’re just adding this text above the last curled bracket (the “}” at the end).

First, we need to ensure our packages are up to date:

exec { "Update packages":
    command => "sudo apt-get update && sudo apt-get dist-upgrade -y",
}

Here’s Apache getting installed:

package { "apache2":
    ensure => "installed",
    require => Exec['Update packages']
}

And, maybe you’ll want to set up something that needs mod_rewrite and a custom site? Add this to your Vagrantfile

config.vm.synced_folder "../Apache_Site", "/etc/apache2/shared_config"

Create a directory called /path/to/your/dev/environment/Apache_Site which should contain your apache site configuration file called “default”. Then add this to your site.pp

exec { "Enable rewrite":
    command => 'a2enmod rewrite',
    onlyif => 'test ! -e /etc/apache2/mods-enabled/rewrite.load',
    require => Package['apache2']
}

file { "/etc/apache2/sites-enabled/default":
  ensure => link,
  target => "/etc/apache2/shared_config/default",
}

So, at the end of all this, we have the following file structure:

/path/to/your/dev/environment
+ -- /Apache_Site
|    + -- default
+ -- /web
|    + -- index.html
+ -- /Vagrant
     + -- /manifests
     |    + -- site.pp
     + -- Vagrantfile

And now, you can add all of this to your Git repository [2], and off you go! To bring up your Vagrant machine, type (from the Vagrant directory):

vagrant up

And then to connect into it:

vagrant ssh

And finally to halt it:

vagrant halt

Or if you just want to kill it off…

vagrant destroy

If you’re tweaking the provisioning code, you can run this instead of destroying it and bringing it back up again:

vagrant provision

You can do some funky stuff with running several machines, and using the same puppet file for all of those, but frankly, that’s a topic for another day.

[1] Vagrant is extended using plugins. There is a list of plugins on this Github Wiki Page. The plugins here can include additional virtual machine back ends (called Providers in Vagrant terminology), and methods of configuring the OS after bootup (called Provisioners), but also anything around defining where to find resources, to define network addresses, even to handle caches and proxies.

[2] If you’re not using Git, you should be! However, you might want to add some stuff to your .gitignore – in particular, Vagrant adds a directory called /path/to/your/dev/environment/Vagrant/.vagrant where it puts the VMs it creates.

A quick update

JonTheNiceGuy Leave a comment

So, my last post ended with me sacking it all in. Fortunately for open source, and my projects in general, a few people stepped up and reminded me why I do this stuff. So, CCHits is still going, and I don’t feel as alone any more with it (which is nice :) ) and CFM development is back on the cards. MOTP-AS is still a bit on hiatus until I get my head around a MVC framework (I’m currently using ZF2 in CFM3).

What is really nice about where I am right now is that I’m learning stuff about the tools I’m using, so it’s not all focusing on stuff which isn’t working, and is instead focusing on the shiny :)

I’ll try and get some posts about Vagrant, Puppet, and ZF2 out as I discover non-trivial stuff about it :)