"LEGO Factory Playset" from Brickset on Flickr

Building Azure Environments in Ansible

Recently, I’ve been migrating my POV (proof of value) and POC (proof of concept) environment from K5 to Azure to be able to test vendor products inside Azure. I ran a few tests to build the environment using the native tools (the powershell scripts) and found that the Powershell way of delivering Azure environments seems overly complicated… particularly as I’m comfortable with how Ansible works.

To be fair, I also need to look at Terraform, but that isn’t what I’m looking at today :)

So, let’s start with the scaffolding. Any Ansible Playbook which deals with creating virtual machines needs to have some extra modules installed. Make sure you’ve got ansible 2.7 or later and the python azure library 2.0.0 or later (you can get both with pip for python).

Next, let’s look at the group_vars for this playbook.

This file has several pieces. We define the project settings (anything prefixed project_ is a project setting), including the prefix used for all resources we create (in this case “env01“), and a standard password used for all VMs we create (in this case “My$uper$ecret$Passw0rd“).

Next we define the standard images to load from the Marketplace. You can extend this with other images, these are just the “easiest” ones that I’m most familiar with (your mileage may vary). Next up is the networks to build inside the VNet, and lastly we define the actual machines we want to build. If you’ve got questions about any of the values we define here, just let me know in the comments below :)

Next, we’ll start looking at the playbook (this has been exploded out – the full playbook is also in the gist).

Here we start by pulling in the variables we might want to override, and we do this by reading system environment variables (ANSIBLE_PREFIX and BREAKGLASS) and using them if they’re set. If they’re not, use the project defaults, and if that hasn’t been set, use some pre-defined values… and then tell us what they are when we’re running the tasks (those are the debug: lines).

This block is where we create our “Static Assets” – individual items that we will be consuming later. This shows a clear win here over the Powershell methods endorsed by Microsoft – here you can create a Resource Group (RG) as part of the playbook! We also create a single Storage Account for this RG and a single VNET too.

These creation rules are not suitable for production use, as this defines an “Any-Any” Security group! You should tailor your security groups for your need, not for blanket access in!

This is where things start to get a bit more interesting – We’re using the “async/async_status” pattern here (and the rest of these sections) to start creating the resources in parallel. As far as I can tell, sometimes you’ll get a case where the async doesn’t quite get set up fast enough, then the async_status can’t track the resources properly, but re-running the playbook should be enough to sort that out, without slowing things down too much.

But what are we actually doing with this block of code? A UDR is a “User Defined Route” or routing table for Azure. Effectively, you treat each network interface as being plumbed directly to the router (none of this “same subnet broadcast” stuff works here!) so you can do routing at the router for all the networks.

By default there are some existing network routes (stuff to the internet flows to the internet, RFC1918 addresses are dropped with the exception of any RFC1918 addresses you have covered in your VNETs, and each of your subnets can reach each other “directly”). Adding a UDR overrides this routing table. The UDRs we’re creating here are applied at a subnet level, but currently don’t override any of the existing routes (they’re blank). We’ll start putting routes in after we’ve added the UDRs to the subnets. Talking of which….

Again, this block is not really suitable for production use, and assumes the VNET supernet of /8 will be broken down into several /24’s. In the “real world” you might deliver a handful of /26’s in a /24 VNET… or you might even have lots of disparate /24’s in the VNET which are then allocated exactly as individual /24 subnets… this is not what this model delivers but you might wish to investigate further!

Now that we’ve created our subnets, we can start adding the routing table to the UDR. This is a basic one – add a route (internet access) from the “protected” network via the firewall. You can get a lot more specific than this – most people are likely to want to add the VNET range (in this case via the firewall as well, except for this subnet (because otherwise, for example, trying to reach will go via the firewall too).

Without going too much into the intricacies of network architecture, if you are routing your traffic between subnets to the firewall, it’s probably better to get an appliance with more interfaces, so you can route traffic across the appliance, rather than going across a single interface as this will halve your traffic bandwidth (it’s currently capped 1Gb/s – so 500Mb/s).

Having mentioned “The Internet” – let’s give our firewall a public IP address, and create the rest of the interfaces as well.

This script creates a public IP address by default for each interface unless you explicitly tell it not to (see lines 40, 53 and 62 in the group_vars file I rendered above). You could easily turn this around by changing the lines which contain this:

item.1.public is not defined or (item.1.public is defined and item.1.public == 'true')

into lines which contain this:

item.1.public is defined and item.1.public == 'true'

OK, having done all that, we’re now ready to build our virtual machines. I’ve introduced a “Priority system” here – VMs with priority 0 go first, then 1, and 2 go last. The code snippet below is just for priority 0, but you can easily see how you’d extrapolate that out (and in fact, the full code sample does just that).

There are a few blocks here to draw attention to :) I’ve re-jigged them a bit here so it’s clearer to understand, but when you see them in the main playbook they’re a bit more compact. Let’s start with looking at the Network Interfaces section!

network_interfaces: |
    {%- for nw in item.value.ports -%}
      '{{ prefix }}{{ item.value.name }}port{{ nw.subnet.name }}'
      {%- if not loop.last -%}, {%- endif -%} 
    {%- endfor -%}

In this part, we loop over the ports defined for the virtual machine. This is because one device may have 1 interface, or four interfaces. YAML is parsed to make a JSON variable, so here we can create a JSON variable, that when the YAML is parsed it will just drop in. We’ve previously created all the interfaces to have names like this PREFIXhostnamePORTsubnetname (or aFW01portWAN in more conventional terms), so here we construct a JSON array, like this: ['aFW01portWAN'] but that could just as easily have been ['aFW01portWAN', 'aFW01portProtect', 'aFW01portMGMT', 'aFW01portSync']. This will then attach those interfaces to the virtual machine.

Next up, custom_data. This section is sometimes known externally as userdata or config_disk. My code has always referred to it as a “Provision Script” – hence the variable name in the code below!

custom_data: |
  {%- if item.value.provision_script is defined and item.value.provision_script != '' -%}
    {%- include(item.value.provision_script) -%}
  {%- elif item.value.image.provision_script is defined and item.value.image.provision_script != '' -%}
    {%- include(item.value.image.provision_script) -%}
  {%- else -%}
    {{ omit }}
  {%- endif -%}

Let’s pick this one apart too. If we’ve defined a provisioning script file for the VM, include it, if we’ve defined a provisioning script file for the image (or marketplace entry), then include that instead… otherwise, pretend that there’s no “custom_data” field before you submit this to Azure.

One last quirk to Azure, is that some images require a “plan” to go with it, and others don’t.

plan: |
  {%- if item.value.image.plan is not defined -%}{{ omit }}{%- else -%}
    {'name': '{{ item.value.image.sku }}',
     'publisher': '{{ item.value.image.publisher }}',
     'product': '{{ item.value.image.offer }}'
  {%- endif -%}

So, here we say “if we’ve not got a plan, omit the value being passed to Azure, otherwise use these fields we previously specified. Weird huh?

The very last thing we do in the script is to re-render the standard password we’ve used for all these builds, so that we can check them out!

Want to review this all in one place?

Here’s the link to the full playbook, as well as the group variables (which should be in ./group_vars/all.yml) and two sample userdata files (which should be in ./userdata) for an Ubuntu machine (using cloud-init) and one for a FortiGate Firewall.

All the other files in that gist (prefixes from 10-16 and 00) are for this blog post only, and aren’t likely to work!

If you do end up using this, please drop me a note below, or star the gist! That’d be awesome!!

Image credit: “Lego Factory Playset” from Flickr by “Brickset” released under a CC-BY license. Used with Thanks!

A bit about my day

I’ve been inspired by this post by one of the leadership team at the company I work for to talk about my working day.

  1. What time do you reach the office each day?
    Around 7:30. I typically work from home, so it’s largely adjusted by what is going on with the family before I get to my desk.
  2. Is your job varied?
    My work schedule is typically the similar sorts of tasks for a week or two, followed by documenting what I’ve done the previous couple of weeks, and then a new task for the next cycle. It means you get long enough to work on one thing before being distracted onto the next thing.
  3. Is your job creative?
    It requires creative and critical thinking to work out why things aren’t working the way you expect them to, or to fix issues for people you’ve agreed to help. It doesn’t typically involve understanding graphical design elements, user experience or visualising customer needs beyond what Visio brings.
  4. What do you spend the majority of your time doing?
    Investigating vendor products, writing Ansible playbooks, supporting consumers of my team’s output and documenting how I’ve integrated the vendor products into the environments I’m working on.
  5. Do you personalize your desk?
    I have a podcasting microphone, an Android plush toy, a lego fencer, two radio transceivers and a cardboard Millennium Falcon. Aside from that, it’s all business.
  6. Would you describe yourself as creative?
    Not particularly. I don’t really have an eye for graphical things and I’m not very good at writing prose. I can grammar check a document well though :)
  7. Do you have any quirky daily rituals?
    Not that I’m aware of. My wife probably disagrees!
  8. Do you tend to work on your own or with colleagues?
    I work from home and don’t work directly with anyone else on the projects I’ve been assigned, but have quite a few meetings with peers and managers during the week, so I don’t feel isolated or like I’m working by myself.
  9. How many hours on average do you work a day?
    I’m contracted for 37 hours a week, but I normally work between 7.5 to 8.5 hours each day. Overall, I’m probably doing about 40 hours. Oops!
  10. Roughly how much time do you spend each day on email? Taking calls? In meetings?
    Email is usually checked for an hour in the morning, then anything critical coming in during the day is handled as it comes in, so probably another 30 minutes or so then. I’m usually on two or three 30 minute online meetings during the day, and there’s probably three or four 15-45 minute calls. Depending on the complexity of the meetings and calls, I might be multi-tasking on other activities during those calls or meetings though.
  11. Do you use social media much for work?
    Not really. I tend to do more by email or phone.
  12. What do you enjoy most about your work?
    Every day is a new challenge, and I get lots of opportunities to help people – whether thats colleagues taking the output of my work, peers looking for guidance in areas I have experience, or being able to help vendors understand how we use their products.
  13. What type of music (if any) do you listen to at work?
    I use Google Music to create instant playlists based on what mood I’m in – usually starting from something like Iron Maiden, Metallica or Linkin Park. If I’m not listening to music, I’ll be listening podcasts, of which I have lots of feeds to select from, or to my local Amateur Radio repeater (GB3WP).
  14. What do you do for lunch?
    Working from home, I tend to eat with my wife and daughter (when she’s not at pre-school).
  15. Do you socialize with work colleagues?
    When we have team meetings, I’ll go out with colleagues.
  16. Are there any tasks (through your career) you’ve been especially glad to get rid of?
    No. My career has been pretty good to me – I started working in retail, specialising in radio products while I was at school and university, when I came back from uni, I went into an office, and learned a bit about sales and financials, and then ended up working for IT firms doing technical support, initially remote support, then on-site, and then for a helpdesk. I progressed from there into server support and then network security. The whole path has gone just right for me, changing at the point when things started to get stale or when I wanted a change. I’ve been exceptionally lucky!
  17. What is your last task of your day?
    Usually finishing up any unsent emails.
  18. How do you like to relax after work?
    I run a yearly conference and a monthly tech meetup. I watch TV with my wife and we play board games.
  19. Do you keep checking email through the evening?
    Nope. I used to, when I had an operational role, but now I don’t need to.
  20. Do you take work projects home with you?
    I work from home, so, technically yes. It’s rare that something from work intrudes into my home life though.
  21. What would you say to your 20-year old self?
    Nothing! 20 year-old me was a bit of an idiot, but without what he did, I wouldn’t be where I am today!
  22. If you could try out any job for a day, what would you choose?
    I like my job, so I don’t think I’d do anything else.
  23. What device did you use to answer these questions?
    Fujitsu E736 running Windows 10
  24. Do you use your own personal device for work?

The importance of saying something – in memory of Lindsey

Today, I said goodbye to a friend. Not a best-of-friends friend, but just a friend, a colleague, a someone-to-chat-with-as-we’ve-both-got-5-minutes friend.

We used to work in the same office, 10 or so years ago, she had been there longer than I and she would teach those who were following the same path she was on. She was a patient and good teacher, I seen to recall. A year or so passed, I moved site, changed roles, but when I came back to the site, I’d try to catch up. It didn’t often work out, but I tried. She had a son a couple of years before me, so I went to her for advice a few times, and it helped knowing she’d followed a similar path.

A few years ago, she was diagnosed with cancer. She did well, fought it off. I only found out near to the end of her treatment, and it didn’t feel right to say anything… after all, we didn’t speak often and, well, I didn’t really know what to say. I mean, what can you say when it’s nearly all over?

I saw she’d beaten it, and I was happy, but again, I’d not said anything when she was fighting it, and so now, what can you say when you’d not said anything when it mattered. I liked some photos and positive messages she’d written on Facebook, I hoped it was enough, to let her know I was happy she was OK.

And then, a few months ago I saw she was back at Christie’s Hospital. She was having more scans. A few weeks later, I was back at the old site and (fortunately) bumped into her. We chatted like old times, shared some war stories about our kids and then she asked if I knew she was back in for treatment. I’m happy to say I was able to say I knew, but I didn’t say much, just that I hoped it all went well and that she was looking good. We didn’t talk for long, but I’m glad we did.

A few weeks ago, the posts from my friend began to change. The posts, still optimistic, were now about a legacy, about pictures, about blankets, and about spending time together. There were pictures of breaks and holidays. I could tell that the outlook had changed… But what could I say? The happy pictures got likes, and again, I hoped that was enough.

And then, I saw a post, my friend had passed away. And it hit me, I’d never really said anything that mattered.

Today, I went to her funeral. Her husband greeted me at the wake, by name. He remembered me, maybe from Facebook, maybe from their wedding. I don’t know. But, he knew me in a room full of people. And eventually, I choked out that sometimes you want to say something, and he replied that there were no words sometimes.

I am glad I went today, if only to know that in some way, I finally managed to say something, even if it wasn’t really enough.

And I realised that, but for the whim of God, or, should you not believe like that, then, on the wheel of chance we call life, this fate that fell on her could fall on any one of us. It could be my wife, my child or even me. It could be my brother, or his family. It could be my neighbour, or another colleague… anyone. And in that case, could I still say nothing? I hope I can find at least something, next time, to say.

Rest in peace, Lindsey. My best wishes to your husband and son, to your sister and mother, and to the friends you have left behind, in a world a little less brighter.

A quick update

So, my last post ended with me sacking it all in. Fortunately for open source, and my projects in general, a few people stepped up and reminded me why I do this stuff. So, CCHits is still going, and I don’t feel as alone any more with it (which is nice :) ) and CFM development is back on the cards. MOTP-AS is still a bit on hiatus until I get my head around a MVC framework (I’m currently using ZF2 in CFM3).

What is really nice about where I am right now is that I’m learning stuff about the tools I’m using, so it’s not all focusing on stuff which isn’t working, and is instead focusing on the shiny :)

I’ll try and get some posts about Vagrant, Puppet, and ZF2 out as I discover non-trivial stuff about it :)

The Apathy of the Lone Coder

I think I might be having a bit of a mid-life crisis. It’ll be my 35th birthday this year, and I’ve started to realise that I don’t really want to do much more of the Open Source’y stuff that I’ve been a part of for the past 10 or so years.

Don’t get me wrong. This isn’t me saying I want to hang up my linux user hat, put away the android phone, wipe the PHP manuals from my kindle or return an HTTP 410 code for everything I’ve ever published… but it’s getting close.

The rot has been setting in for some time.

November 2011 was the “first birthday” of CCHits.net – I’d planned to have my site-wide re-write of the whole code base ready for the birthday, but frankly, I’d massively underestimated the amount of work involved, so it wasn’t ready for November. As it was, a critical failure on my web host prompted me to “make” the rewrite work in April – nearly half a year after it was supposed to be in by. I’m not at all happy with the site layout, the way the tracks are build, the lack of adoption of the service by any other podcasters than the three who currently submit to the site (no criticisms there for anyone else, just a frustration really) and, well, the fact it never really achieved the vision I had for it.

In April, I helped to organise UCubed – a one day unconference about Linux and Open Source [1], held at MadLab, Manchester. We put less effort into organising it than we had the last few times, I pretty much wimped out on the day, taking my son to his swimming lesson (which meant leaving two hours after the event started, and returning an hour before it finished), and after the event, I felt like all I’d done was go to get the refreshments.

In July and August, I pulled a lot of 2 and 3AM finishes to get CampFireManager ready for OggCamp. I had some solid support from a guy called Jack who committed a load of great code to the project, plus loads of encouragement from the organisation team for OggCamp, the big day came, and, well, let’s just say there were issues. Quite a lot of issues really. I missed all of both mornings of talks because I was fire fighting those issues, and on the second day, I was held up as an example of “why not to code something instead of just doing it”. I had a top notch PHP engineer [2] sitting next to me while I was looking through issues, and even though I’ve gone through the theory of how the site works with her before, she couldn’t get her head around it. OK, I was skimming through the code pretty fast and I know most of it like the back of my hand so I knew roughly where code had gone and was going to next but still… code is code, right? Not if it’s crap code with unusual structure, insufficient testing, incomprehensible logic and, well, it’s just crap…

Before OggCamp, I inadvertently became the project lead for something I still don’t fully understand (although I’m a lot closer on it, to be fair): MOTP-AS. An implementation of the Mobile One Time PIN algorithm, written in PHP, tied up to a FreeRadius Server with a pretty web UI to give something a bit like RSA SecurID Authentication Manager server. Essentially, I made some suggestions on how to improve the code, and was told “Well, actually, we were pretty much going to kill off the project after the next release – do you want to take it over?” and I, in hindsight, stupidly said “Oh, OK”. I said that from October, I’d have “loads” of time, and was going to re-write the code base using Object Oriented principals, was going to roll in Unit Testing, PHPDocumentor and, theoretically, move to using a sensible framework to render the whole thing.

The hindsight thing I mentioned there? On the 28th August, my father passed away. I’ve not really talked about it much on Social Media. It’s a pretty hard thing to do, as it may mean airing an awful lot of dirty laundry as a result, but I guess the outcome of that was that I’ve been spending a lot more time away from my home, staying instead at my fathers home where I have been clearing it to sell it, and when I’ve not been away from home, I’ve wanted to spend more time with Jules and Daniel.

The first couple of trips down to my Dad’s house were on the train. I tried to break open a text editor and start turning out reusable PHP which I could form into something in MOTP-AS, but let’s be serious about this, it was like trying to read a book in the same circumstances – you just keep reading the same page over and over again, but nothing “right” comes out the other side. I’ve not had the enthusiasm to even start to look at that project since then.

Everyone I was working with – CCHits, CampFireManager, MOTP-AS – all knew I was offline, and would be “for some time”, but the funk that set in on that train hasn’t shifted yet, and I still can’t work out if it’s something to do with my Dad, or just the fact that I’m not really feeling the code right now.

At a recent PHPNW session, Lorna said (although I am paraphrasing) that most of my bad practices come from a lack of exposure to other PHP developers, and that working as part of a team towards something would help. My day job has nothing to do with coding (and there’s no scope to bring it into my role, and the few times I’ve tried to bring it in, it’s caused me more issues with my work than if I hadn’t) and 5% to do with open source software (the 5% is due to the OS that many of the devices we support are RedHat, BSD or Solaris based). I don’t want to, and can’t afford to make a career change now (aside from anything else, I still love my job, especially what I’m doing at the moment) to get that experience, and I’m getting closer and closer to burning out on the projects I’m involved in – just because there’s no one else who understands it like I do… which is sad.

When I do start to code in the evenings, what I tend to do is think of something I’d like to write (yep, starting a new project will fix *everything* Jon!), open my IDE, try and work out what I want to learn to use this time, and start reading the documentation for it… and not actually start working on the project. And then 2 hours have passed, I’ve done nothing, and frankly I could do with going to bed.

So, how do I beat this apathy folks. Is there anyone out there who can help?

I think if I’ve not sorted something out by June, I’ll close down CCHits.net. It’s been a great blast, but I’m so nervous of something going wrong with the system and it collapsing like a pack of cards… which is a real shame as HPR [3] have just said they’ll be running the daily shows in their Icecast server when “real” feeds aren’t being streamed, that and I love discovering, or re-discovering the music which is played through the system.

Likewise, I think I’ll probably try and find someone to hand CFM over to during OggCamp this year, and if I can’t find someone to hand it over to, I’ll shut it down. Again, it’s been fun, but I don’t need 2 months of sleepless nights and 2 days of sheer panic for something which ultimately could be replaced by a sheet of paper and some post-it notes.

Of all of the projects I’ve mentioned, the MOTP-AS part is most likely to be something of use to me in my day job (which was, in fact, how I came across it… for our lab network), so I might make more of an effort with that, but again, I really can’t see me being happy with it at the end of it all.

[1] It used to be about more than that, but frankly, it’s what it turned into.
[2] Plug for that top notch PHP engineer who, fortunately for me, was happy (or if not actually happy, appeared to be happy enough) to be an observer, a person to bounce ideas off, a muse and cheerleader (sort-of) for those two days of hell – http://LornaJane.net
[3] HackerPublicRadio.net – a podcast network made up from individual posts by the community.

Hacking Flattr support into StatusNet 0.9.5

So, recently I’ve really got into Flattr (from flattr.com) and I wanted to add a flattr icon to my StatusNet instance.

I’m running StatusNet 0.9.5 (hopefully Dreamhost will upgrade us to 1.x soon!) but for now, adding Flattr to that page means adding it to the Site Notice (the box in the top corner).

Now, to make that work, I amended the config table, which limits all the config items to 255 characters. I changed the Value column to “Text” (was Varchar(255)) and then in the admin page, added some random bumpf into the “Site Notice” page (at http://example.com/admin/sitenotice) and then found that same entry in the database (SELECT * FROM `config` WHERE `section` LIKE ‘site’ AND `setting` LIKE ‘notice’). I then amended what I’d written in there before to be the code I got from the “Get Button” page (both <script> and <a> tags from https://flattr.com/submit), and selected “Create Button Manually”.

I then refreshed by StatusNet front page, and tada! I’ve got a Flattr button. Now, if *only* I could get that as a plugin for my site.

Another e-mail to my MP #debill #AndrewGwynneMP

Dear Andrew [Gwynne MP],

I just noticed that, according to both this website:
http://debillitated.heroku.com/ and this one:
http://www.didmympshowupornot.com/ that you were not able to attend
the debate around the Digital Economy Bill.

Given that over 20,000 people are recorded as having written to their
MP, and over 23,000 twitter messages (since the 17th March) were sent
about the subject, and less than 4% of their MPs attended, I’m
slightly worried about whether this bill is being given appropriate

Will you reassure me that this bill is considered important enough to
your constituents (like me) that you will be asking the house to be
dropped until after the election, when it can receive the proper
scrutiny it deserves, or do you not feel that it’s important for your
constituents for you to be involved in the debate around a law which
permits the government to modify copyright without proper debate, and
which also enables “rights holders” to arrange the disconnection of a
whole household from participating on the Internet, where those
“rights holders” allege (and are never required to be proven in court)
file sharing has occurred.

Yours, in hope,

Jon Spriggs

Posted via email from Jon’s posterous