Using Amazon an EC2 instance as an off-site CronJob

I run the CCHits.net website, and part of the day-to-day tasks that running the site entailed was the daily show creation which involved creating some text-to-speech audio for insertion into the podcasts. As I run the website on shared hosting, to which I didn’t have full access to the OS, I couldn’t just install Festival on the platform, and for whatever reason (I never did figure out what went wrong), I couldn’t build Festival to run on the shared host either.

Until “The Big Rewrite” (the capitals are totally worth it), I’d been doing the text-to-speech on my home server, but frankly, I’m on DSL, which meant I needed to set up Dynamic DNS, I had to be sure the server was always up (and it wasn’t!), etc, etc, etc. While I was looking into why I couldn’t get Festival to build, someone said “Well, why not just use EC2 to do it”.

After nearly a year of faffing about trying to make the …….. stupid thing work (as is testified by the draft in this very blog called “How I built my Audio Stack for CCHits”), I finally decided to spin up an EC2 instance for just this one task.

Now, I’m not the greenest guy on the block – hell, I drive 45 minutes into work each day, but I figured, why keep an EC2 instance running all the time, when I only need it for less than 20 minutes each day, so I did some reading, and found a post on making EC2 do the hard work for you, using the scalable computing APIs, but frankly, all I actually needed was the code to make it spin up, run the task and shut down again, especially as with using his methods, I’d have needed to either create an AMI image, or download the festival voice files each time… at around 100Mb. Not good. I ended up using the stuff I did know (bash scripting, cron tasks) and ditching the stuff I didn’t (AMI files, scalable computing API). I may revisit this later to do it the way he said instead. We’ll see :)

So, here’s the crack.

Create your EC2 image. It doesn’t need to do anything fancy yet – just boot up and keep running. You’ll do some tweaks later. Make a note of the instance number – it’ll probably start i- and then 8 or so hexedecimal digits, like this: i-12345678.

On your shared web host, download the EC2 API tools. According to this page, the API tools are available from here. The first link of those two is unlikely to change for a while, the second, maybe more so. You’ll need to make sure you have Java installed on that web host.

Once you’ve got the tools, you’ll need to create an X.509 certificate and key pair. See this page to find instructions. It was pretty straightforward.

So, you now have in, let’s say for sake of argument, your home directory:

  • /home/user/ec2-api-tools-x.x.x.x # The directory containing all the EC2 API tools
  • /home/user/ec2-keys/pk-{32[A-Za-z1-0]}.pem
  • /home/user/ec2-keys/cert-{32[A-Za-z1-0]}.pem

Also, you have java in /usr/bin/java.

Create the following script also in /home/user/ec2-api-tools-x.x.x.x – I called it ec2-wrapper.

#!/bin/bash
export EC2_HOME=/home/user/ec2-api-tools-x.x.x.x
export JAVA_HOME=/usr
export EC2_KEY=/home/user/ec2-keys/pk-{32[A-Za-z1-0]}.pem
export EC2_CERT=/home/user/ec2-keys/cert-{32[A-Za-z1-0]}.pem
${EC2_HOME}/bin/$* -K ${EC2_KEY} -C ${EC2_CERT

Obviously, you should change your paths to match what you have. What this script does is to add the X.509 certs to every EC2 request, plus adds the appropriate java and EC2_HOME paths to the script before running it.

I set up a CRON job (using crontab -e) to schedule the regular startup of the instance. Here’s the entry from my crontab:

#M   H  DoM Mth DoW  Command (Regular Crontab columns)
30   1   *   *   *   /home/user/ec2-api-tools-x.x.x.x/ec2-wrapper ec2-start-instances i-12345678
30   2   *   *   *   /home/user/ec2-api-tools-x.x.x.x/ec2-wrapper ec2-stop-instances i-12345678

So, this runs the start task at 30 minutes past 1am, local server time, and the stop task at 30 minutes past 2am. The second one there is just to be on the safe side, as we’ll try to shut down the box once it’s finished processing anyway. This way, the maximum time you’ll be billed for is 1 hour of time each day.

I then logged into my EC2 machine, and created, then tweaked the script from the earlier blog post (the scalable computing one).

#!/bin/bash -x
exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
#
# This code is licensed under an Apache License - for the original 
# code and license, please see the footer of this script.
#
# !!!IMPORTANT!!!
# Edit this file and change this next line to your own email address:
#

EMAIL=user@example.com

# Get some information about the running instance
instance_id=$(wget -qO- instance-data/latest/meta-data/instance-id)
public_hostname=$(wget -qO- instance-data/latest/meta-data/public-hostname)

# Wait 5 minutes in case we want to get in to stop this from doing stuff
Sleep 300

if [ -f /home/ubuntu/donotrun ]
then
  exit 0
fi

# Send status email
/usr/sbin/sendmail -oi -t -f $EMAIL <<EOM
From: $EMAIL
To: $EMAIL
Subject: Running EC2 Scripts

== Making sure everything is up to date ==

`cd /home/ubuntu/website-rewrite && su -c "git pull" ubuntu 2>&1`

== Running the scheduled task ==

`php /home/ubuntu/website-rewrite/scheduled_task.php`

== Notes ==

This email was generated on the EC2 instance: $instance_id

If the instance is still running, you can monitor the output of this
job using a command like:

  ssh ubuntu@$public_hostname tail -1000f /var/log/user-data.log

EOM

# Give the email some time to be queued and delivered
sleep 300 # 5 minutes

if [ -f /home/ubuntu/shutdownwhendone ]
then
  shutdown -h now
fi

exit 0

########################################################################
# For more information about the original version of this code see:
#   http://alestic.com/2011/11/ec2-schedule-instance
# The original code and its license are available on github:
#   https://github.com/alestic/demo-ec2-schedule-instance
########################################################################

So, with that, I get a script which runs on schedule, on an EC2 platform, I get a confirmation e-mail it’s running. It shuts itself down, and hopefully, keeps on trucking :)

SOPA

So, today was the “great blackout”.

I have to be honest, although I participated in the blackout [1], I’ve not really been much impacted by the blackout today. I tried to access wikipedia a couple of times, and drew a blank, well… ish. I tried to follow up a link to a profile on identi.ca, and got a black page. I tried to get a link to a podcast I listen to, and got a redirect to another site.

So, how do I feel it went? As a guy who lives in the UK, I think it was really interesting. I saw more comments about the loss of Wikipedia than I did about SOPA, I saw that the BBC did a news story on Wikipedia being down, and I had a really hard time explaining why it was an issue to my wife (to be fair she said “you always go into a long rambling explanation, why don’t you just summarize it in a sentence for me” – I said “It’ll might take down the blog site you spend a lot of time reading”, and she said “Oh, OK”).

I saw a lot of people posted a link to The Oatmeal’s SOPA blackout comic. I’d link to it, but it’s not exactly work safe, and you might be browsing this site, and don’t want to see a koala making sweet love to a donkey, or Oprah and Jesus on a jetski in space! Er, so anyway.

I don’t know. I hope it had much more of an impact in the US, but I think really, most people would have been going “WTF? Wikipedia is down… now how are we going to find out what the 23rd episode of the Transformers TV series was about” than about anything to actually do with SOPA, and I think people who did take an interest are going to be more like my wife (“how is this actually *really* going to impact me looking at lolcats” – not actual quote) than me (“OMG, The stinking AMERICAN GOVERNMENT are going to take down my podcast, my blog and all my code” – nearly actual quote).

[1] CCHits.net had a redirect on the main page to http://sopastrike.com/, and the daily exposure show was just Doris (actually the Festival voice “cmu_us_clb_arctic_clunits”) saying some stuff I programmed into her. The nearly-actual quote above was actually something like “This law, if enacted could take down not just cchits.net, but my personal blog, open source code I have written, my e-mail server, my authentication systems, and this is just my content. I use a shared web server, and every other customer on the same server could also be affected if any single user posts material they do not own the copyright to. Simply put, the law the American government wants to enact would destroy everything that is good about the internet.” There was more than that, but like my wife said, I waffle a lot. Sorry. See, I’m doing it again. BAH!

What I Did Last Week (ending 2012-01-15)

Monday 9th: I had a frantic morning before work, moving the last few bits onto our bed before the decorations started. When I got to work, it was a quiet work day, which I was glad of after a hectic weekend. It was quiet in more ways than one – my bluetooth headset died, so I ordered a duplicate as a replacement. I nearly bought a Sony Ericsson LiveView at the same time as the price was nearly what I had previously considered paying, until I read the reviews! Had a couple of emails from my dad which were similar to the conversations I’d had with him Friday Night (see last week for details!) Sent a very emotional response that didn’t seem to make a dent. Never mind. Once Daniel had gone to bed, Jules and I curled up in the bombsite of our living room and watched a couple of programs before going to bed. I then finished “Daemon” 5*s, and I bought the sequel “Freedom TM” also by Daniel Suarez. I started reading “Cyberpunk Stories” by William King which I picked up today. While I was reading, Daniel sat up in his sleep and then fell over, Jules and I think he might be a sleep walker, which will be fun – even more cause for fitting stair gates! One final chat with Dave Lee about Powerline adaptors before going to sleep.

Tuesday 10th: I booked Wednesday off work, and plan to make the most of it. Forgot to set my Out Of Office messages before leaving for the day, so needed to dip back into my e-mail once I was home. Jules and I watched TV for a bit before going to bed, and I realised that I’d got about 6 tracks to submit to cchits (three from a google alert I’ve got set up for CC licensed music, one from a recommendation on statusnet, and two that were played on the Crivins podcast), however, I’m staying offline this week in the evenings as much as possible, so they’ll have to wait until I’m releasing my self-imposed blockade. Posted the powerline adaptors to Dave Lee, hope they help him out! Finished “Cyberpunk Stories” 2*s and then read “Freedom TM” 5*s which is an amazing twist on the “Daemon” book… I think if the author ever collaborates with someone like Cory Doctorow, for the futurisms, or better yet, Charles Stross, for taking a scary concept and making it both funny and deeply understandable… well, let’s just say I want to see what else comes from this author. While I was reading, Daniel sleep-sat-up again twice, bumping his head the second time around. Oh well, maybe he’s just disturbed by all the decorating stuff.

Wednesday 11th: Day off!! Yey!! Early start then off to Head-over-heels, a soft play centre near Hanforth Dean. Daniel had his morning nap while I was driving us there, so Jules jumped out and did some shopping while I listened to “How governments have tried to block Tor” video from 28C3 which was at the recommendation of a colleague. Ironic I’d not heard of it, given that I’d done a presentation on Tor at the first OggCamp. Spent 2hrs at Head-over-heels including getting lunch then went to the Trafford Centre. Daniel had his afternoon nap en-route to the Trafford Centre, so I dropped Jules off at the Trafford Centre and carried on listening to the Tor talk. Very little I’d not heard before, but great to have it in context. Picked up the full Father Ted box set, plus the “X-Men: 1st Class” DVD after having heard some great reviews. Daniel got a new pram book from Waterstones, which he then spent the whole rest of the time there flicking through it. Jules picked up two new lego board games “Sunblock” and “Race 3000“. I swear we’ve nearly got all of the lego games now. Got home to find wet-paint walls in the dining room and wet skirting boards in the lounge, so Daniel and Jules or I spent the whole evening except for dinner in his room. Once he went to bed, Jules and I played a couple of games each of Sunblock and Race 3000, then Jules went to sleep. I bought three new books for my Kindle “Beloved Weapon” by Jonathan A. Price, “The Windup Girl” by Paolo Bacigalupi and “Empire State” by Adam Christopher. I started, and am 38% through “Beloved Weapon”, and although there’s a fair bit of gratuitous and graphic sex scenes, it’s a pretty good superhero story thus far. While I’ve been typing this up, Dave Lee’s been in touch to say that the Powerline Ethernet adaptors I sent him had arrived and we did some diagnostics around why the throughput was low. At the same time, Daniel’s been stirring a lot again. I’ve had to help him back down from sitting up twice already tonight, and that’s not counting the times he’s sorted himself out. Oh well. Today overall has been a good day.

Thursday 12th: really rubbish night, with Daniel waking after sitting up and falling and banging his head, and then not settling for over an hour. Got into work to be told that while I’d been off, a serious issue had occurred with something I’d implemented (which didn’t make sense, as we’d created accurate documentation based on the data I’d entered into the devices in question). Those two items together sent me into a bit of a spin and left me questioning myself for most of the day. Finished more-or-less on time. When I got home, Jules asked me to lower the matress on the cot as Daniel is getting proficient at pulling himself up on the side. Doing this meant I also fixed the under-cot-drawer which had been broken within a couple of weeks of us building it (before Daniel was born!). We played with Daniel until his bed time, and then once he was down, we had Chinese take-away and then played Upwords until Jules was tired. After Jules went to sleep, I finished “Beloved Weapon”. It had, frankly, a rubbish end and barely rated the 2*s I gave it. Personally, I think it paid too much attention to the sex and physical relationships between the characters than it did to any background, non-physical relationships or plot. I probably wouldn’t read anything else by this author. Started to read “Empire State”, but only managed a chapter before I got too sleepy.

Friday 13th: Yet another disturbed night. Jules had promised to take care of Daniel all night, but about 2 hours after I’d fallen asleep, he woke up screaming. Jules couldn’t calm him down and asked for some help. I went in and finished calming him down to just crying, while Jules went downstairs and got him some Calpol. After he’d taken it, he settled well, and Jules put him back down. He then woke up at about 6 and woke us both up. When I got into work, I was covering a collegue while he was on a customer visit, in addition to my normal accounts. One of my normal accounts scheduled a two-hour conference call starting at 12:00 and then at 16:00 (when I was due to be leaving) I got a call from my collegue’s account asking me to implement an urgent change for him. So I ended up leaving 40 minutes late. When I got home, the work downstairs is all finished, so Jules had pushed the sofas out to the edges so Daniel could play, and for the first time in a week, we both sat down with him and played too. When he went to bed, we watched some tv and then I spoke to my brother on the phone for 30 minutes. Jules went to bed at 9:15 and I listened to the live Bugcast show from my phone (I’d not had my laptop out at all this week) until the end of the show, when Dave proposed doing a Google Hangout. Out came the laptop and headphones and I ended up going to bed at about midnight. No reading tonight, straight to sleep – busy day tomorrow.

Saturday 14th: up at 6:45, breakfast and then I went out to get a radiator cover. When I got back, I built it and then loaded up the car for the tip. Jules’ Mum and Dad arrived and started emptying our room into the dining room. When Daniel woke up, the work started in ernest… By 1pm, we’d got most of the stuff down we were ready for, so I took Daniel to his swimming lesson, via the tip and a nap enroute to the class. Lesson went well, but I cut my foot during the lesson (banged it against the steps) but didn’t notice until I got out and, while I was getting dressed, I was bleeding all over the place. I swear, it looked like there had been a massacre in there! Asked the teacher for a plaster, filled out the accident form and went home. Daniel had his dinner then while I put him to bed, Jules nipped out to pick up dinner from the supermarket. We had pizza and watched “Take Me Out” (a guilty pleasure), then the after-show follow up on ITV2. It’s funny how obvious it is that they must pre-record it weeks if not months in advance, as a scandal broke out about the show after last week’s episode, that one of the contestants of this game show used to be a prostitute… And she was completely cut out of the show, even though she’s there in all the video clips and is there in the after show, but they don’t talk to her. Sad really. Early night.

Sunday 15th: not only is my cut foot stinging like crazy, but the ankle on my other foot has gone gout’y again. Aargh. Jules let me have a lay in until 9 AM then while I put Daniel down for his nap, Jules went shopping. When she got back, we all went to do the food shopping for the week, something we’ve not done together for months. Daniel was hungry while I sorted out paying for our purchases, so she fed him in the cafe, then when I got in there I went over all funny. Jules bought me a sandwich then I went out to the car and felt really sick. Jules dropped me off at home and then drove Daniel around so he could have a sleep without disturbing me. When they got home I was feeling much better, so I put the shopping away while Jules sorted out dinner. After dinner, I bathed Daniel, we played with him before bottle and bed and then Jules and I snuggled up on the sofa. We watched TV for an hour or so, and then went to bed. I’ve not read anything tonight, but I did catch up on some social network updates I’ve missed. Tired though, do an early night for me!

What I Did Last Week (ending 2012-01-08)

Inspired by Dan Lynch’s “Weekly Rewind” series, I thought I’d try and document some of what’s happened to me over the past week… you never know, I might even be able to keep on doing these! :)

Monday 2nd: Bank Holiday. De-christmasified the house. Earlier than I’d have liked, but it was a compromise, as Jules wanted to strip the house on Boxing Day. Dave Lee from TheBugCast adds the first CCHits.net “extra” show to his feed – Review of the tracks played on the site in 2011. It had been played on the live show on the 30th Dec.

Tuesday 3rd: Back to work, and catching up with collegues about what happened over the break. Trying and failing to compile Festival for CCHits. Called out… but not for anything sensible – just a license request for a customer – told them to get back in touch during the day.

Wednesday 4th: Discussed at length the differences between Access and Excel with my brother. Convinced I had the same discussion in 2004. He wants to start with a ToDo list which he can filter to show customers or management. Once he’s figured that out, I’m going to talk to him about separating data from presentation in a web app with a database backend. Maybe! Bought and read “Boltman” by Eric Quinn Knowles. It’s a little bit like “Kick Ass” vs. Scientology. I rate it 4*’s. Started reading “Under the Amoral Bridge” by Gary A. Ballard which I’d bought in November.

Thursday 5th: Fixed a Morse Code Keyboard for Android for a collegue (unfixed typos in the code for openbracket, and switched equals and hyphen characters) – developer hasn’t fixed issues since March 2010, so I’ve e-mailed the author to offer my patches, and failing that, I’ll consider a fork. Realised I’d fallen very far behind on my Android development suite – aside from anything else, the version of Eclipse I’ve got installed needed updating! I read to the end of “Under the Amoral Bridge” (4*s) and discovered it’s part 1 in a trilogy. Bought the compilation version of the trilogy, so I’m now reading book 2.

Friday 6th: Took down the decorations at work, then gladly handed over “On Call” for another week to a collegue, discovered they were on leave. Queue frantic phone calls and texts to make sure he knew he was on call. At 3:30 get asked if I could cover him for the night as it’s his wife’s 40th birthday. How can I say no? No calls, fortunately! Listen live to TheBugCast. Dave and Caroline have streaming issues and I get a call from my dad claiming his computer has been compromised, as he can’t log into GMail. Prove there’s no issues there, believe that is the end of it… In the aftershow, I mention the code acadamy site, and then explain some of the concepts of Javascript to another of the listeners, which is good :) end up going to bed at 2am.

Saturday 7th: visit from Jules’ uncle’s brother to discuss getting some decorating done. He can start on Monday. Queue full-scale panic as he’s doing both downstairs rooms! All CD’s and DVD’s not in storage, boxed up moved upstairs and unpacked… now need re-organising! All pictures, bottles, games, Daniel’s toys and books now in our bedroom! Argh! Daniel’s first swimming lesson with the new teacher (new term, and the franchise we go to is growing – which is good!) New teacher is nice and the class is still just 3 children and parents. Good stuff. While I’m in the swimming class, I get two emails from my dad. First is a scattergun one, saying “my email has been compromised, if you get a dodgy mail, let me know” and the second is to check whether he has been compromised. As we’re in full-scale panic, I’ve left that particular issue to my brother to deal with. I suspect drinking-related-paranoia is at the core of this. Never mind! We finish the bulk of the moves before Daniel’s bed time, and then Jules’ Mum and Dad come around to babysit so we can enjoy a meal up the road. The restaurant isn’t licensed, so neither of us can drink, and hasn’t got card processing facilities yet. I nip out after I’ve finished to get enough money to pay. We’ll definitely be back there! Early bed, but then I read all the rest of book 2 and most of book 3 of “The Bridge Chronicles”.

Sunday 8th: Earlyish start. Finish moving the last furniture around for the decorating and start to re-wire the entertainment corner, including unpatching the server where the shows are generated for CCHits.net just before the shows are about to be run (stupid UTC offsets!), resulting in two stinking CRON mails from CCHits complaing about the lack of shows (repatched and run). Run Jules to Halfords to pick up her new bike with child seat. Home and one last sprint around the house, then lunch, and out again! Jules to the shops and me to get Daniel to sleep before heading to a friend’s new house for a tour, games and then dinner. Home at 5, Daniel in bed, 6:45 and the furniture we couldn’t shift while he was awake, away for 7:30. Books in bed for 8. I finish “…Chronicles” (5*s) and then at the recommendation of @nybill, start “Daemon” by Daniel Suarez. At 11pm, stop reading (40% through the book) and start writing this review. 11:40 go to sleep!

Hacking Flattr support into StatusNet 0.9.5

So, recently I’ve really got into Flattr (from flattr.com) and I wanted to add a flattr icon to my StatusNet instance.

I’m running StatusNet 0.9.5 (hopefully Dreamhost will upgrade us to 1.x soon!) but for now, adding Flattr to that page means adding it to the Site Notice (the box in the top corner).

Now, to make that work, I amended the config table, which limits all the config items to 255 characters. I changed the Value column to “Text” (was Varchar(255)) and then in the admin page, added some random bumpf into the “Site Notice” page (at http://example.com/admin/sitenotice) and then found that same entry in the database (SELECT * FROM `config` WHERE `section` LIKE ‘site’ AND `setting` LIKE ‘notice’). I then amended what I’d written in there before to be the code I got from the “Get Button” page (both <script> and <a> tags from https://flattr.com/submit), and selected “Create Button Manually”.

I then refreshed by StatusNet front page, and tada! I’ve got a Flattr button. Now, if *only* I could get that as a plugin for my site.

Transfer my files using SFTP and SCP only?

A colleague today asked for some guidance around setting up an SFTP and SCP only account on a RedHat based Linux machine.

I sent him a collection of links, including one to the CopSSH project, and he implemented the code on that link, but then struggled when it didn’t work.

Aside from the fact the shell wasn’t copied into /etc/shells (which wasn’t disastrous, but did mean we couldn’t reuse it again later), it was still returning an error on each load.

Doing some digging into it, and running some debugging, I noticed that pscp (the PuTTY SCP) tool uses the SFTP subsystem rather than the SCP command to upload files, so we need to also check that the SFTP server hasn’t been called, instead of the SCP command, and also the SCP command needs to be corrected.

Here follows a script, complete with comments. Personally, I’d save this in /bin/sftponly, created and owned by root, and set to permissions 755 (rwxr-xr-x). Then, set the shell to this for each user which needs to do SFTP or SCP only.

#!/bin/bash
# Based on code from http://www.itefix.no/i2/node/12366
# Amended by Jon Spriggs (jon@sprig.gs)
# Last update at 2011-09-16

# Push the whole received command into a variable
tests=`echo $*`

# Set up a state handler as false
isvalid=0

# Test for the SFTP handler.
# The 0:36 values are the start character and length of the handler string.
if [ "${tests:0:36}" == "-c /usr/libexec/openssh/sftp-server" ]; then
  # Set the state handler to true
  isvalid=1
  # Configure the handling service
  use=/usr/libexec/openssh/sftp-server
fi

# Test for the SCP handler.
if [ "${tests:0:6}" == "-c scp" ]; then
  # Set the state handler to true
  isvalid=1
  # Configure the handling service
  use=/usr/bin/scp
fi

# If the state handler is set to false (0), exit with an error message.
if [ "$isvalid" == "0" ]; then
  echo "SCP only!"
  exit 1
fi

# Run the handler
exec $use $*

Quick Fix for Apache CVE-2011-3192 – Ubuntu/Debian

Please note – Apache have released a fix to this issue, and as such the below guidance has now been superseded by their fix.

I have been aware of the Apache web server issue for the last few days, where an overly wide range is requested from the server, leading to a crash in the server. As a patch hasn’t yet been released by Apache, people are coding their own solutions, and one such solution was found at edwiget.name.

That fix was for CentOS based Linux distributions, so this re-write covers how to do the same fix under Debian based distributions.
Read More

Getting my head around coding Object Orientated PHP

I’ve been writing two open source projects over the last couple of years. My code has never really been particularly great, but I’ve been trying to learn how to improve my code and over the last few months, I’ve been really trying to polish up my coding skills.

A few months back, I attended a series of fantastic sessions at PHPNW about using Unit Testing, PHP CodeSniffer and phpDocumentor, and how these can be incorporated into Object Orientated code (or in fact, requiring Object Orientated code to implement them).

So, I went back into my main projects and started to look at how I could fix the code to start adopting these tools.

So, the first thing I needed to do was to start thinking about the structure. CCHits.net (which is the “big project” I’ve been working on recently) has several chunks of data, and these are:

User
Track
Artist
Show
ShowTrack
Vote
Chart

Each of these have been broken down into three “things” – The Object itself, a “Broker” which finds all the relevant objects, and a class to create new items, so let’s start with a user class. We’ll define a few properties in the initial class creation.

class UserObject
{
    protected $intUserID = 0;
    protected $strOpenID = "";
    protected $strCookieID = "";
    protected $sha1Pass = "";
    protected $isAuthorized = 0;
    protected $isUploader = 0;
    protected $isAdmin = 0;
    protected $datLastSeen = "";
}

By setting these as protected, it stops me from directly setting or accessing these variables from outside of the class – instead I want to do it from a function, so let’s add those in (I’ll just do one – assume these will be copied on to the rest of the values).

class UserObject
{
    protected $strOpenID = "";

    function set_strOpenID($strOpenID = "") {
        if ($strOpenID != $this->strOpenID) {
            $this->strOpenID = $strOpenID;
        }
    }

    function get_strOpenID() {
        return $this->strOpenID;
    }
}

In the set_ functions, we already do a little bit of error checking here (is the value already set to this?) but we could add other things like, on the integer items, is it actually an integer, with the boolean values ($is[SOMETHING]), make sure it’s set to 1 or 0 (or true/false).

Now, let’s add some documentation to this:

/**
 * CCHits.net is a website designed to promote Creative Commons Music,
 * the artists who produce it and anyone or anywhere that plays it.
 * These files are used to generate the site.
 *
 * PHP version 5
 *
 * @category Default
 * @package  CCHitsClass
 * @author   Jon Spriggs 
 * @license  http://www.gnu.org/licenses/agpl.html AGPLv3
 * @link     http://cchits.net Actual web service
 * @link     http://code.cchits.net Developers Web Site
 * @link     http://gitorious.net/cchits-net Version Control Service
 */
/**
 * This class deals with user objects
 *
 * @category Default
 * @package  Objects
 * @author   Jon Spriggs 
 * @license  http://www.gnu.org/licenses/agpl.html AGPLv3
 * @link     http://cchits.net Actual web service
 * @link     http://code.cchits.net Developers Web Site
 * @link     http://gitorious.net/cchits-net Version Control Service
 */
class UserObject
{
    /**
     * This is the Setter function for the strOpenID value
     *
     * @param string $strOpenID The value to set
     *
     * @return void There is no response from this function
     */
    function set_strOpenID($strOpenID = "") {
        // Do stuff
    }
}

So, let’s make it do stuff with the database. Firstly, we need to set up a connection to the database. I’ll be using PDO (PHP Database Object, I think) to set up the connection (I’ll show why in a minute).

So, here’s another class – Database, this time it’s using a singleton factory (which is to say, while it may exist many times in the code, it’ll only ever have one connection open at once) – note it’s got hard-coded authentication details here – this isn’t how it actually is in my code, but this way it’s a bit more understandable!

class Database
{
    protected static $handler = null;
    protected $db = null;

    /**
     * This function creates or returns an instance of this class.
     *
     * @return object $handler The Handler object
     */
    private static function getHandler()
    {
        if (self::$handler == null) {
            self::$handler = new self();
        }
        return self::$handler;
    }

    /**
     * This creates or returns the database object - depending on RO/RW requirements.
     *
     * @return object A PDO instance for the query.
     */
    public function getConnection()
    {
        $self = self::getHandler();
        try {
            $self->rw_db = new PDO('mysql:host=localhost;dbname=cchits', 'cchits', 'cchits', array(PDO::MYSQL_ATTR_INIT_COMMAND => "SET NAMES utf8"));
            return $self->rw_db;
        } catch (Exception $e) {
            echo "Error connecting: " . $e->getMessage();
            die();
        }
    }
}

So, now let’s create yet another class, called GenericObject. This will be re-used in all of the “Object” classes to perform our main database calls. Try to move as much code up to your highest level objects – so you could have a “validateBoolean($value)” function that is used any time you set or get a boolean value here… we even mentioned something like this above!

class GenericObject
{
    protected $arrDBItems = array();
    protected $strDBTable = "";
    protected $strDBKeyCol = "";
    protected $arrChanges = array();

    /**
     * Commit any changes to the database
     *
     * @return boolean Status of the write action
     */
    function write()
    {
        if (count($this->arrChanges) > 0) {
            $sql = '';
            $strDBKeyCol = $this->strDBKeyCol;
            $values[$strDBKeyCol] = $this->$strDBKeyCol;
            $values = array();
            foreach ($this->arrChanges as $change) {
                if ($sql != '') {
                    $sql .= ", ";
                }
                if (isset($this->arrDBItems[$change])) {
                    $sql .= "$change = :$change";
                    $values[$change] = $this->$change;
                }
            }
            $full_sql = "UPDATE {$this->strDBTable} SET $sql WHERE {$this->strDBKeyCol} = :{$this->strDBKeyCol}";
            try {
                $db = Database::getConnection();
                $query = $db->prepare($full_sql);
                $query->execute($values);
                return true;
            } catch(Exception $e) {
                return false;
            }
        }
    }

    /**
     * Create the object
     *
     * @return boolean status of the create operation
     */
    protected function create()
    {
        $keys = '';
        $key_place = '';
        foreach ($this->arrDBItems as $field_name=>$dummy) {
            if ($keys != '') {
                $keys .= ', ';
                $key_place .= ', ';
            }
            $keys .= $field_name;
            $key_place .= ":$field_name";
            $values[$field_name] = $this->$field_name;
        }
        $full_sql = "INSERT INTO {$this->strDBTable} ($keys) VALUES ($key_place)";
        try {
            $db = Database::getConnection();
            $query = $db->prepare($full_sql);
            $query->execute($values);
            if ($this->strDBKeyCol != '') {
                $key = $this->strDBKeyCol;
                $this->$key = $query->lastInsertId();
            }
            return true;
        } catch(Exception $e) {
            return false;
        }
    }

    /**
     * Return an array of the collected or created data.
     *
     * @return array A mixed array of these items
     */
    function getSelf()
    {
        if ($this->strDBKeyCol != '') {
            $key = $this->strDBKeyCol;
            $return[$key] = $this->$key;
        }
        foreach ($this->arrDBItems as $key=>$dummy) {
            $return[$key] = $this->$key;
        }
        return $return;
    }
}

Any protected functions or variables are only accessible to the class itself or classes which have been extended from it (referred to as a child class). We need to make some changes to our UserObject to make use of these new functions:

class UserObject extends GenericObject
{
    // Inherited Properties
    protected $arrDBItems = array(
        'strOpenID'=>true,
        'strCookieID'=>true,
        'sha1Pass'=>true,
        'isAuthorized'=>true,
        'isUploader'=>true,
        'isAdmin'=>true,
        'datLastSeen'=>true
    );
    protected $strDBTable = "users";
    protected $strDBKeyCol = "intUserID";
    // Local Properties
    protected $intUserID = 0;
    protected $strOpenID = "";
    protected $strCookieID = "";
    protected $sha1Pass = "";
    protected $isAuthorized = 0;
    protected $isUploader = 0;
    protected $isAdmin = 0;
    protected $datLastSeen = "";

    function set_strOpenID($strOpenID = "") {
        if ($this->strOpenID != $strOpenID) {
            $this->strOpenID = $strOpenID;
            $this->arrChanges[] = 'strOpenID';
        }
    }
}

Notice I’ve stripped the “phpDoc” style comments from this re-iteration to save some space! In the real code, they still exist.

Now we have an object we can work with, let’s extend it further. It’s probably not best practice, but I find it much more convenient to create new objects by extending the UserObject into a new class called NewUserObject. Notice once we’ve set our database items, we run the create(); function, which was previously defined in the GenericObject class.

class NewUserObject extends UserObject
{
    public function __construct($data = "")
    {
        if (strpos($data, "http://") !== false or strpos($data, "https://") !== false) {
            $this->set_strOpenID($data);
        } elseif ($data != "") {
            $this->set_sha1Pass($data);
        } else {
            if (isset($_SERVER['HTTP_X_FORWARDED_FOR'])) {
                $cookie_string = $_SERVER['HTTP_X_FORWARDED_FOR'];
            } else {
                $cookie_string = $_SERVER['REMOTE_ADDR'];
            }
            $cookie_string .= $_SERVER['HTTP_USER_AGENT'];
            $cookie_string .= $_SERVER['HTTP_ACCEPT_LANGUAGE'];
            $cookie_string .= $_SERVER['HTTP_ACCEPT_ENCODING'];
            $cookie_string .= $_SERVER['HTTP_ACCEPT_CHARSET'];
            $this->set_strCookieID(sha1sum($cookie_string));
        }
        $this->datLastSeen = date("Y-m-d H:i:s");
        $_SESSION['cookie'] = sha1($cookie_string);
        return $this->create();
    }
}

Using the code $user = new NewUserObject($login_string); we can create a new user, but how about retrieving it.

This is where the reason I’m loving PDO comes into play. See, before PDO, when you did a database request, you might have had something like this:

$db = mysql_connect("localhost", "root", "");
if ($db == false) {
    die ("Failed to connect to the Database Server");
}
if (! mysql_select_db("database")) {
    die ("Failed to select the database");
}
$intUserID = mysql_real_escape_string($_GET['intUserID']);
$sql = "SELECT * FROM users WHERE intUserID = '$intUserID' LIMIT 1";
$qry = mysql_query($sql);
if (mysql_errno() > 0) {
    echo "Failed to make Database Call: " . mysql_error();
} else {
    if (mysql_num_rows($qry) == 0) {
        echo "Failed to retrieve record 1";
    } else {
        $row = mysql_fetch_array($qry);
    }
}

Now, let’s assume you’re looking for a few users? You need to do more of those mysql_real_escape_strings and mysql_num_rows() and mysql_fetch_array()s to get your data out – it’s far from clean and clear code.

How about this instead?

try {
    $db = Database::getConnection();
    $sql = "SELECT * FROM users WHERE intUserID = ? LIMIT 1";
    $query = $db->prepare($sql);
    $query->execute(array($_GET['intUserID']));
    $row = $query->fetch();
} catch(Exception $e) {
    echo $e;
    $row = false;
}

The most complicated bit here is that you’re having to prepare your SQL query and then tell it what to get. The reason we do that is that PDO will automatically ensure that anything being passed to it using the ? is sanitized before passing it into the query. If you look back at the GenericObject class we created earlier, it uses something like this there too, except there (if you work out what it’s doing) it prepares something like INSERT INTO users (intUserID) VALUES (:intUserID); and then executes it with the values like this: array(‘:intUserID’=>1) and with this you can have some very complex statements. Other code you might spot while mooching through CCHits (if you decide to) looks like this:

$sql = "UPDATE IGNORE votes SET intTrackID = ? WHERE intTrackID = ?; DELETE FROM votes WHERE intTrackID = ?";
$query = $db->prepare($sql);
$query->execute(array($intNewTrackID, $intOldTrackID, $intOldTrackID));

By wrapping it all up in a try/catch block, you can get your error dumped in one place, including a stack trace (showing where the issue turned up). You don’t need to check for mysql_num_rows – if the row doesn’t exist, it’ll just return a false. Sweet.

Where it gets REALLY nice, is that if you swap $row = $query->fetch() with $object = $query->fetchObject(‘UserObject’); and it’ll create the object for you!

So, now, with the function getUser (visible at https://gitorious.org/cchits-net/website-rewrite/blobs/master/CLASSES/class_UserBroker.php) it’ll try to return a UserObject based on whether they’re using OpenID, Username/Password or just browsing with a cookie… and if it can’t, it’ll create you a new user (based on the above criteria) and then return you that UserObject.

The last thing to add, is that I wrapped up all the nice phpDocumentor, PHP CodeSniffer functions, plus wrote a script to check for missing or incorrectly labelled functions across a suite of classes. These sit in https://gitorious.org/cchits-net/website-rewrite/trees/master/TESTS if you want to take a look around :)

EDIT 2011-08-25: Correcting some errors in the code, and to adjust formatting slightly.
EDIT 2012-05-05: Changed category, removed the duplicated title in the top line, removed some whitespace.

VNC in Ubuntu 11.04 with Unity

I recently bought myself a new laptop. Sometimes though, I want to check something on it on a rare occasion when I’ve not taken it with me. In comes VNC. Under Ubuntu 11.04, turning on VNC support is pretty straight forward.

To turn on VNC, go to the power icon in the top right corner (I think they call it the “Session Menu”, but it looks like a power button to me) and select “System Settings”. Under the “Internet and Network” heading, is an option called “Remote Desktop”. Click on that. Tick the top two boxes “Allow other users to view your desktop” and “Allow other users to control your desktop”. Tick the box “Require the user to enter this password” (and enter a password) and “Configure network automatically to accept connections”. Untick “You must confirm each access to this machine” and select “Only display an icon when there is someone connected”. Close it.

Now, try connecting to your device, and see what happens. I had some issues with Compiz elements not rendering correctly, and found a few hints to fix it. The first says to turn on the “disable_xdamage” option. It says to use gconf-editor, but I’m SSHing in, so I need to use gconftool-2 as follows:

gconftool-2 --set "/desktop/gnome/remote_access/disable_xdamage" --type boolean "true"

Personally, I only want to ever connect over OpenVPN to this, so I added the following:

gconftool-2 --set "/desktop/gnome/remote_access/network_interface" --type string "tun0"

You may wish to only ever access it over SSH, in which case replace “tun0” with “lo”

Now, I next made a big mistake. I followed some duff guidance, and ended up killing my vino server (I’m still not sure if I was supposed to do this or not), but to get it back, I followed this instruction to restart it. I had to tweak it a little:

sudo x11vnc -rfbport 5901 -auth guess

Once you’ve started this, tunnel an extra port (5901) to your machine, start VNC to the tunnelled port, and then go back through the options above. Exit your VNC session to the new tunnelled port, and then hit Control+C on the SSH session to close that x11vnc service.