Tag: Hints and Tips

Using riot.im as a bridge to Freenode

JonTheNiceGuy Leave a comment

Over the last few months, I’ve been using Riot.im (the client for Matrix.org) as my primary IRC client, and access to other end-to-end-encrypted chats.

A few weeks ago I decided I wanted to use my “normal” IRC nickname on Freenode, so looked into how to do it. It’s surprisingly easy, but there are a few gotchas.

Making Matrix know your password

First of all you need to message the IRC bridge bot and tell it your nickserv password: !storepass chat.freenode.net MyComplexPassword

Next, you need to chat with NickServ directly and authenticate with it: identify MyNormalNick MyComplexPassword

Lastly, you go back to your chat with the bridge bot, and tell it your nickname: !nick chat.freenode.net MyNormalNick

If, in the process of doing this, you find you can’t log in as yourself, message NickServ and tell it to release your account from being protected: release MyNormalNick MyComplexPassword

Setting topics

While the bridge bot will let you set a topic (!cmd TOPIC #channel Something) this didn’t really work for me, so instead, I use ChanServ to do it for me: topic #channel My New Topic

Just remember that you must have ops granted to you for that channel through ChanServ to be able to make such changes.

You can also set modes for people, or the channel, ban people, voice or op them through ChanServ, just send the command help in the chat to ChanServ for more guidance!

And, if you’re stuck, feel free to come ask for help! I’m (predictably) JonTheNiceGuy!

Building a Dual boot machine running Ubuntu 17.04 and Windows 10 with full-disk encryption

JonTheNiceGuy 4 Comments

This post has been revised since it was initially published on 31st March due to errors found in the resulting build. It was also missing details on the shared data drive between the two machines, so has been amended to include that.

** WARNING ** This works for me – it might not for you!

The outcome of this build will leave you with the following:

Boot up, go through the VeraCrypt bootloader, enter a password for Windows, or press escape to load the Grub bootloader where you will boot (K|L|X|)Ubuntu(| Mate| Gnome).

The Windows environment will be encrypted with VeraCrypt, an open source Full Disk Encryption technology, while the Linux environment will be encrypted using Luks. The shared volume (between Windows and Linux) will be encrypted with VeraCrypt.

PLEASE BE AWARE THAT ANY WINDOWS 10 UPGRADES WILL FAIL TO APPLY AS IT WILL NOT RECOGNISE THE VERACRYPT FILE SYSTEM! To resolve this, decrypt the Windows volume, perform the upgrade, re-encrypt it, then transfer the new recovery ISO image to the boot volume, following the method below. Yes, this will take some time. No, you don’t need to decrypt the data volume. Yes, you can use that data volume to shunt the ISO image around.

LATE EDIT 2020-01-06: I’ve just spotted a link to this post over on Level1Techs. In that post, someone asked if the broken upgrades is still a thing. Turns out that since I wrote this in 2017, it’s not been fixed. Now, I should stress, I’ve stopped using this layout as I went all-Linux on that machine, but… it might work for you now?! Also, shout out to 92aceshigh for referencing this post, and glad something I wrote helped you! ☺

Step 1:Create your partition table

My partition table, for a 320GB Disk looks (roughly) like this:

Partition 1: 20GB – Linux /Boot (ext2, plus space for ISO files for random booting)
Partition 2: 60GB – Windows C:\ (NTFS VeraCrypt)
Partition 3: 72GB – Linux Physical Volume (LVM PV, Luks Encrypted)
– logical volume 1: 16Gb Swap (Linux Swap)
– logical volume 2: 60Gb Linux (ext4)
Partition 4: 156GB – Shared Volume (NTFS, VeraCrypt)

I performed this using GParted in the Gnome Live image using the GParted. Some rational here:

  1. The first partition also allows me to add other ISOs if I want to boot them.
  2. I have 4GB RAM, this gives me some extra space to allow me to hibernate, but also… 4Gb. Ugh.
  3. I then split my Linux and Windows partitions into two equal parts.

Step 2: Use Cryptsetup to format the disk

The following steps need to be run as root.

sudo -i

Step 2a: Format the partitions as LUKS

cryptsetup luksFormat -y -v /dev/sda3

Step 2b: Open the LUKS volume

cryptsetup luksOpen /dev/sda3 lvm-pv

Step 2c: Create the LVM Physical Volume over the LUKS volume

vgcreate vg00 /dev/mapper/lvm-pv

Step 2d: Define the LVM Logical Volumes over the LVM Physical Volume

lvcreate -n lv00_swap -L 16G vg00       # Define 16GB Swap Space
lvcreate -n lv01_root -l +100%FREE vg00 # Define the rest of vg00 as /

LEAVE YOUR TERMINAL OPEN

Step 3: Install your Linux distribution.

Note that when you perform your install, when you get to the partitioning screen, select “Manual”, and then pick out the following volumes:

/dev/mapper/vg00-lv01_root = ext4 formatted, mount point: /
/dev/mapper/vg00-lv00_swap = swap
/dev/sda1 = ext2, format, mount point: /boot

Select the boot volume of /dev/sda. But wait, I hear you say, Windows has a well know history of nuking Grub partitions… Well, we’ll sort that in a bit…

DON’T EXIT THE LIVE SESSION ONCE THE INSTALL HAS FINISHED (select “Continue Testing”).

Step 4: Make your machine actually able to boot

Go back to your terminal session.  It should still be logged in as root. We need to re-mount all the partitions…

Step 4a: Mount your volumes

mount /dev/mapper/vg00-lv01_root /target
mount /dev/sda1 /target/boot
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /target$i; done

Step 4b: Swap to the “Target” filesystem

chroot /target

Step 4c: Setup your volumes to prompt for cryptographic keys

echo "LinuxRoot UUID=`blkid | grep sda3 | cut -d\\\" -f2` none luks" > /etc/crypttab

Step 4d: Update the boot volume to use these changes

update-initramfs -u

Step 4e: Ensure Grub is also installed to the MBR for testing

grub-install --force /dev/sda1
chattr +i /boot/grub/i386-pc/core.img
update-grub

The first part installs grub to the boot position, even though it doesn’t like it, and the second forces the core file to be unchangeable… I’m not exactly sure of the impact of this, but it’s the only way to do the next part of this process. The last bit makes sure that you’ve got the latest grub config files installed.

Step 5: Reboot and test

Just check to make sure the machine boots OK!

You should have a booting Ubuntu derivative with an encrypted file system. Awesome.

Now let’s install Windows!

Step 6: Install Windows and Veracrypt.

You should boot from your install media, when you get to the partition selector, there should only be a single NTFS partition for it to use. Use it.

Install the latest version of Veracrypt from https://veracrypt.codeplex.com/

Once it’s installed, go to System, Encrypt System Partition/Drive, “Normal” system encryption, Windows System Partition, Multi-Boot (accept the warning), Boot Drive “Yes”, Single Disk, “Non-Windows Boot Loader” – No, and then… let it go through all the rest of the steps. There will be one stage where it asks you to create a rescue disk. Just save it for later. Once the encryption settings are collected, it will do a test (which is basically just rebooting to the boot loader, having you put in your password and going back into Windows), and then let it start performing the encryption.

Once the encryption finishes, reboot the computer, enter the decryption password and test it boots to Windows OK. Then reboot it again and press escape instead of putting in the password. It will boot to your Ubuntu system.

So, there you have it. One Dual-Boot system with encrypted disks everywhere!

Step 7: Setting up the shared volume.

After you’ve got the Ubuntu and Windows volumes sorted out, next we need the shared data volume to be organised. You’ll need Veracrypt for Ubuntu. Use the following to install the Veracrypt package for Ubuntu:

sudo add-apt-repository ppa:unit193/encryption
sudo apt-get update
sudo apt-get install veracrypt

Once that’s installed, boot back into Windows and create a new volume – perhaps V: for Veracrypt, or E: for Encrypted – your choice, but make sure you create it using the same password that you used for the Windows partition.

Format this new volume with either NTFS or FAT32 so that you can mount it under either operating system. I chose NTFS.

Now, you need to go into Veracrypt’s Settings menu, and select “System Encryption Settings”. Tick “Cache pre-boot authentication password in driver memory” (be aware, this means that if your machine is compromised when powered up, the password could be recovered), then OK. This may prompt you to accept the UAC at this point.

Next, with the mounted volume selected, go to the “Favorites” menu, and choose “Add to System Favorites”. In the screen which comes up, select the box under “Global Settings” which says “Mount system favorite volumes when Windows starts (in the initial phase of the startup procedure)”. There will be a warning about passwords that appears. Click OK.

You may, at this point, want to move certain aspects of your Windows desktop (e.g. the “My Documents” location) to the new mounted drive.

On the Linux OS, become root, with sudo, and then add the following lines to your crontab:

@reboot mkdir -p /shared_storage 
@reboot veracrypt --text --non-interactive --fs-options=uid=1000,gid=1000,umask=0077 --password="YOURSUPERSECUREPASSWORD" /dev/sda4 /shared_storage

These assume that your login user’s ID is 1000 (you can check that by running the command “id” as your logged in user), that you want to use “/shared_storage” as the mount point (it stops Ubuntu treating it as a “Mountable Volume” if it’s not in your home directory and not in /mnt or /media). These options also mean that only that user (and root) can access any of the files in that partition (although, it is only you on this laptop… right?), which means you can safely use it for any files which check user permissions before allowing you to access them (e.g. SSH keys). I then set up a symbolic link to /home/MYUSERACCOUNT/Documents into the /shared_storage/Documents directory, and /home/MYUSERACCOUNT/.ssh into the /shared_storage/SSH_Keys directory.

Citataions

The following list of resources helped me out when I was struggling with what to do next! They may not be canonical sources, but they helped.

  1. http://thesimplecomputer.info/full-disk-encryption-with-ubuntu – This is what got me started on this little journey!
  2. http://askubuntu.com/questions/161689/how-do-i-get-grub2-to-boot-a-truecrypt-encrypted-mbr – How to add the Veracrypt recovery disk to your Grub boot partition. Note, I do it slightly differently to this now.
  3. http://askubuntu.com/questions/711801/i-deleted-files-in-boot-now-cant-boot-linux – I may have done this. It tells you how to put all your important files back for booting purposes :)
  4. https://www.youtube.com/watch?v=Z1yWbBIqh1o – Walk through of installing Veracrypt to Windows 10. I used this to see some of the terms after I’d already installed Veracrypt. I don’t quite follow the same route as him though.
  5. https://www.linux.com/blog/how-full-encrypt-your-linux-system-lvm-luks – Using LVM inside Luks for the full-Linux disk (this was why I’ve re-written this post)
  6. https://wiki.archlinux.org/index.php/Dm-crypt/Encrypting_an_entire_system – Some details around how the Luks stuff all works

I may or may not have reinstalled Windows and Kubuntu about 20 times during this process, cursing myself for starting the whole damn process off in the first place!!!

Working with complicated template data UserData in Ansible

JonTheNiceGuy Leave a comment

My new job means I’m currently building a lot of test boxes with Ansible, particularly OpenStack guests. This means I’m trying to script as much as possible without actually … getting my hands dirty with the actual “logging into it and running things” perspective.

This week, I hit a problem standing up a popular firewall vendor’s machine with Ansible, because I was trying to bypass the first-time-wizard… anyway, it wasn’t working, and I couldn’t figure out why. I talked to my colleague [mohclips] and he eventually told me that I needed to use a template, because what I was trying to do was too complicated.

But, damn him, I knew that wasn’t the answer :)

Anyway, I found this comment on a ticket, which lead me to the following… if you’re finding that your userdata: variable in the os_server module of Ansible isn’t working, you might need to wrap it up like this:

userdata: |
  {%- raw -%}#!/bin/bash
  # Kill script if the pipe fails
  set -euf -o pipefail
  # Write everything from this point on to Syslog
  echo " == Set admin credentials == "
  clish -c 'set user admin password-hash {% endraw -%}{{ default_password|password_hash('sha512') }}{%- raw -%}' -s
  {% endraw %}

Note that, if you have a space before your variable, use {% endraw -%} and if you’ve a space after it, use {%- raw %} as the hyphen means “ditch all the spaces before/after this command”.

One to read or watch: “Programming is Forgetting: Toward a New Hacker Ethic”

JonTheNiceGuy Leave a comment

Here is a transcript of a talk by Allison Parrish at the Open Hardware Summit in Portland, OR. The talk “Programming is Forgetting: Toward a New Hacker Ethic” is a discussion about the failings of the book “Hackers” by Steven Levy. Essentially, that book proposed (in the 80’s) a set of ethics for Hackers (which is to say, creative programmers or engineers, not malicious operators). Allison suggests that many of the parables in the book do not truly reflect the “Hacker Ethic”, and revises them for today’s world.

Her new questions (not statements) are as follows:

  • Who gets to use what I make? Who am I leaving out? How does what I make facilitate or hinder access?
  • What data am I using? Whose labor produced it and what biases and assumptions are built into it? Why choose this particular phenomenon for digitization or transcription? And what do the data leave out?
  • What systems of authority am I enacting through what I make? What systems of support do I rely on? How does what I make support other people?
  • What kind of community am I assuming? What community do I invite through what I make? How are my own personal values reflected in what I make?

This is a significant re-work of the original “Hacker Ethic“, and you should really either watch or read the talk to see how she got to these from the original, especially as it’s not as punchy as the original.

I’d like to think I was thinking of things like these questions when I wrote CampFireManager and CCHits.

Using Expect to SFTP a file

JonTheNiceGuy Leave a comment

Because of technical limitations on a pair of platforms I’m using at work, I am unable to set-up key-based SFTP or SCP to transfer files between the pair of them, so I knocked together this short script using the TCL based Expect language.


#!/usr/bin/expect
set arg1 [lindex $argv 0]
set arg2 [lindex $argv 1]
set arg3 [lindex $argv 2]
set timeout 1000
spawn sftp "$arg2"
expect {
yes {
send "yes\r"
exp_continue
}
ass {
send "$arg3\r"
exp_continue
}
sftp {
send "put $arg1\r"
expect {
100% {
send "quit\r"
exp_continue
}
}
}
}

view raw

upload.exp

hosted with ❤ by GitHub

There’s no error checking here, which isn’t great, but as a quick-and-dirty script to SFTP files to a box which needs the password each run… it works! :)

GPG Encrypting files using a keyserver

JonTheNiceGuy Leave a comment

Another “at work” post!

I’ve been generating files which need to be distributed via a file server, but need to be encrypted using GPG (the open source PGP application). Rather than managing keys for a large number of users, instead, I have a text file with the user names in, and a batch file. Please see the below gist for details :)

Installing Symantec Endpoint Protection (SEP) on Ubuntu 14.04

JonTheNiceGuy Leave a comment

At work we use Symantec Endpoint Protection, and in a lab, I was asked to confirm whether we could install it on our Ubuntu 14.04 servers. This took a few hops to get it installed, so I figured, I’d publish how I got it working, to save some other poor soul the trouble :)

Firstly, add the webupd8team’s Java PPA and update the repository cache: sudo add-apt-repository ppa:webupd8team/java && sudo apt-get update

This gives you the ability to install the Java 8 installer: sudo apt-get install oracle-java8-installer

This should download the install files, but for some reason, I was struggling to download it (the install script seems to struggle with downloading the actual .tar.gz file from Oracle), so I manually followed the link to http://download.oracle.com/otn-pub/java/jdk/8u77-b03/jdk-8u77-linux-x64.tar.gz, accepted the license, and placed the file in /var/cache/oracle-jdk8-installer/ and then re-ran the above apt-get install line.

— Note: This above issue was because I was running a caching proxy, which somehow doesn’t play nicely with this script. Turn off your proxy – should be all good :)

Next I had to install the Java Cryptography Extension which I got from the Java SE page. I placed this file in /tmp/jce_policy-8.zip (the filename is the one Oracle use) and replaced the files in /usr/lib/jvm/java-8-oracle/jre/lib/security with the ones from the extracted archive with this line: cp -b /tmp/UnlimitedJCEPolicyJDK8/*.jar /usr/lib/jvm/java-8-oracle/jre/lib/security.

The SEP client also has a dependency on the 32bit version of GLibc. I installed this with sudo apt-get install libc6-i386

I was then, finally, able to install the SEP client by unpacking the installer zip file, and running sudo bash install.sh -i from the path I’d unpacked the zip file in.

Not very complicated, I guess!

— Sources:

  • https://ubuntuincident.wordpress.com/2011/04/14/install-the-java-cryptography-extension-jce/
  • http://www.linuxquestions.org/questions/linux-newbie-8/how-to-install-32-bit-glibc-2-9-or-later-on-64-bit-ubuntu-12-04-a-4175413667/
  • http://www.webupd8.org/2012/09/install-oracle-java-8-in-ubuntu-via-ppa.html

Setting up a Google Play Music uploader for Linux Servers

JonTheNiceGuy 8 Comments

THIS POST HAS BEEN SUPERCEEDED.

I like having an online music server. At home, I use a Logitech Media Server (formerly Squeezebox Server) [1] and run my several O2 Jogglers around the house with the “Squeezeplay OS” [2] images to play music from that server, but when it comes to an Android tablet (or Android phone), there’s not that joined-up thinking from Logitech (although you can just about cobble it together using a few 3rd party apps [3]), but what I do like is the Google Play Music service.

Google Play Music [4] was the first product from the Google Play team after they rebranded to “Google Play”, and I pretty quickly got interested in it. I installed the Google Play Uploader [5] on my home server, and uploaded all my music (apparently, I’m up to 12,000 tracks, but I think there are some duplicates there!) but what to do about the rest of the family? Well, until just recently, it didn’t matter. Jules has no interest in playing music on her phone or tablet, and Daniel, well, he’s 3-and-a-bit.

Since pretty early on, he’s been all over our tech – initially just using whatever apps we had installed on Jules’ and my phones, then Jules’ tablet (I was, and still am, pretty cautious about him using my tablet as it’s an Asus TF300T [6] which cost quite a lot of money, and I keep toying with the idea of installing some other OS – like Firefox OS [7] or Ubuntu Touch [8] on there and can’t if it’s wrecked), and now he’s pretty comfortable with browsing YouTube or the Play Store, although he knows not to click on adverts and can’t install anything that costs anything.

In the last couple of months, since he’s been really learning how to spell, he’s been asking how to spell the names of his favourite films (“Oliver!”, “Frozen” and “The Polar Express” primarily) to get the film clips up in YouTube, or to play snippets from the soundtracks of the films… which got me thinking. I can’t really do much about the films (not yet at least!), but perhaps I could set up Google Play Music with the soundtracks he listens to… but I already have one account sync’ed with Google Play Music from my media server…. how do I get his stuff set up on there?

Essentially, Google Play Music Uploader is a GUI application that can’t be started in the CLI (which kinda makes sense from a simplicity perspective), but as I’m already running one instance of the application on my media server, I can’t start up a second one, so what do I do?

Well, as it turns out, there’s actually a python library for interacting with Google Play Music’s upload and download applications called “gmusicapi” [9], and this couples with a really nice wrapper gives me a CLI utility I can run in a CRON job on my media server.

The wrapper is called gmusicapi-scripts [10], which contains 3 utilities – gmdownload.py, gmsync.py and the key to this – gmupload.py.

You need to install a few libraries. On my Ubuntu 14.04 system, I needed to run:

sudo apt-get install python-pip avconv-tools
sudo pip install gmusicapi
sudo pip install docopt

Once you’ve got this, you can get the tools themselves like this:

git clone https://github.com/thebigmunch/gmusicapi-scripts.git

This will give you a folder called gmusicapi-scripts in which is gmupload.py – the first time you run it, it’ll ask you to visit a web page in order to register the client. Click “OK” to approve the library having access to Google Play Music. This is a pretty spartan page, and ends up with a grey text box containing a string. Copy the contents of that box back into your terminal, and hey-presto, you get it working…

Well, sort of. For me, because I’d not set Daniel up with Google Play Music yet, I needed to set up his account first. I didn’t realise this (I thought, just going to the web page the script points you to will get you access, but it doesn’t because they need to vet which country you’re accessing from…) and the script didn’t tell me that (it just kept saying “Not Subscribed” [11])

Anyway, once it’s done you run

~/gmusicapi-scripts/gmupload.py /path/to/file.mp3

( 1/1) Successfully uploaded /path/to/file.mp3

If you’ve got multiple users, you can rename ~/.local/share/gmusicapi/oauth.cred to ~/.local/share/gmusicapi/SomeOtherName.cred and then run

~/gmusicapi-scripts/gmupload.py -c SomeOtherName /path/to/file.mp3

Subsequent passes will prompt you to authenticate the next account as you go, and then you can rename them as appropriate.

[1] Logitech Media Server (formerly Squeezebox Server): http://www.mysqueezebox.com/download
[2] “Squeezeplay OS”: http://birdslikewires.co.uk/articles/squeezeplay-os
[3] a few 3rd party apps: I got it working on my TF300T by combining https://play.google.com/store/apps/details?id=de.bluegaspode.squeezeplayer and https://play.google.com/store/apps/details?id=de.cedata.android.squeezecommander
[4] Google Play Music: https://play.google.com/music
[5] Google Play Uploader: https://play.google.com/music/listen?u=0#/manager
[6] TF300T: http://en.wikipedia.org/wiki/Asus_Transformer_Pad_TF300T
[7] Firefox OS: https://www.mozilla.org/en-US/firefox/os/
[8] Ubuntu Touch: http://www.ubuntu.com/tablet
[9] “gmusicapi”: https://github.com/simon-weber/Unofficial-Google-Music-API
[10] gmusicapi-scripts: https://github.com/thebigmunch/gmusicapi-scripts
[11] “Not Subscribed”: I raised a bug, but the lead developer said it seemed obvious to him that you have to set it up first… not disputing that for the first pass, but a nice message would have been good :) https://github.com/thebigmunch/gmusicapi-scripts/issues/22

Some notes about Ethernet over Power

JonTheNiceGuy Leave a comment

I messed around a bit with my network tonight, in order to set set up my Ethernet-over-power (AKA Powerline Networking), and I figured out some things which, while they may not be useful to many of you, this is a bit of a prompt for the next time around.

1) The manager application runs under Windows only (although apparently, there are github repositories where you can get and build a linux application which even lets you set QoS aka Quality Of Service and other such fun things – I’ve not tried them, so I can’t recommend them). If you’ve got more than a matched pair of these, then you’ll need to run the application. I didn’t try running it in a virtual machine – I kept the supplied Windows OS from when I bought this machine specifically for purposes like this.

2) Not all models reset in the same way. If you can’t get them all to reset, connect to them with a CAT5 cable, go to the “Privacy” tab, select “Public network” which will reset it to “PowerLineAV”, and then select “Local computer”. You should then be able to browse across them all.

3) Not all models come with a “password” (sometimes referred to as a DEK). In this case, you also have to plug into these devices to set up their security. If they do have a password, it’ll be entirely in upper case, and even though the application shows numeric characters, in the 4 devices I received, they were all alphabetic-only strings of 16 characters, separated by hyphens.

4) Once you’ve got them all set to “PowerLineAV”, typed the passwords in for the models which have them, you can now set a community wide network password. This could be used to set up several logical segments, but realistically, it’s going to be one flat network :)

I can’t think, offhand, of anything else I need to say right now, but it’s been pretty interesting setting this up, so… hope you enjoyed it!

Starting EC2 instances using PHP

JonTheNiceGuy Leave a comment

I run a small podcast website called CCHits.net. It runs on Dreamhost because they offer unlimited storage and bandwidth, but while it’s a great service for storage, it’s not really useful for running a batch process because long running processes are killed regularly (in my case, building the cchits podcasts on a daily basis).

As a result, I built an EC2 instance which I trigger every day using a cronjob. Previously, I used the “AWS CLI tools”, but as this uses a Java Virtual Machine, it was taking an awful lot of resources just to spin up an instance, and Dreamhost kept killing the task off. As a result, I found the AWS PHP SDK, and coded up this little snippet to spin up the EC2 instance.