This is a quick post to remind me in the future… and is based on this official help page.
execute restore vmlicense tftp yourlicense.lic 192.0.2.1
This works around what you can do if you accidentally apply a license you’ve already applied somewhere else :)
CCHits was recently asked to move it’s media to another host, and while we were doing that we noticed that many of the Monthly shows were broken in one way or another…
Cue a massive rebuild attempt!
We already have a “ShowRunner” script, which we use with a simple Vagrant machine, and I knew you can use other hypervisor “providers”, and I used to use AWS to build the shows, so why not wrap the two parts together?
Firstly, I installed the vagrant-aws plugin:
vagrant plugin install vagrant-aws
Next I amended my Vagrantfile with the vagrant-aws values mentioned in the plugin readme:
Vagrant.configure(2) do |config|
config.vm.provider :aws do |aws, override|
config.vm.box = "ShowMaker"
aws.tags = { 'Name' => 'ShowMaker' }
config.vm.box_url = "https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box"
# AWS Credentials:
aws.access_key_id = "DECAFBADDECAFBADDECAF"
aws.secret_access_key = "DeadBeef1234567890+AbcdeFghijKlmnopqrstu"
aws.keypair_name = "TheNameOfYourSSHKeyInTheEC2ManagementPortal"
# AWS Location:
aws.region = "us-east-1"
aws.region_config "us-east-1", :ami => "ami-c29e1cb8" # If you pick another region, use the relevant AMI for that region
aws.instance_type = "t2.micro" # Scale accordingly
aws.security_groups = [ "sg-1234567" ] # Note this *MUST* be an SG ID not the name
aws.subnet_id = "subnet-decafbad" # Pick one subnet from https://console.aws.amazon.com/vpc/home
# AWS Storage:
aws.block_device_mapping = [{
'DeviceName' => "/dev/sda1",
'Ebs.VolumeSize' => 8, # Size in GB
'Ebs.DeleteOnTermination' => true,
'Ebs.VolumeType' => "GP2", # General performance - you might want something faster
}]
# SSH:
override.ssh.username = "ubuntu"
override.ssh.private_key_path = "/home/youruser/.ssh/id_rsa" # or the SSH key you've generated
# /vagrant directory - thanks to https://github.com/hashicorp/vagrant/issues/5401
override.nfs.functional = false # It tries to use NFS - use RSYNC instead
end
config.vm.box = "ubuntu/trusty64"
config.vm.provision "shell", path: "./run_setup.sh"
config.vm.provision "shell", run: "always", path: "./run_showmaker.sh"
end
Of course, if you try to put this into your Github repo, it’s going to get pillaged and you’ll be spending lots of money on monero mining very quickly… so instead, I spotted this which you can do to separate out your credentials:
At the top of the Vagrantfile, add these two lines:
require_relative 'settings_aws.rb'
include SettingsAws
Then, replace the lines where you specify a “secret”, like this:
aws.access_key_id = AWS_ACCESS_KEY
aws.secret_access_key = AWS_SECRET_KEY
Lastly, create a file “settings_aws.rb” in the same path as your Vagrantfile, that looks like this:
module SettingsAws
AWS_ACCESS_KEY = "DECAFBADDECAFBADDECAF"
AWS_SECRET_KEY = "DeadBeef1234567890+AbcdeFghijKlmnopqrstu"
end
This file then can be omitted from your git repository using a .gitignore file.
A few months ago I was a guest on The Ubuntu Podcast, where I mentioned that I use Streisand to terminate my VPN connections. I waffled and blathered a bit about how I set it up, but in the end it comes down to this:
streisand.vm.network "public_network", bridge: "eth0", ip: "192.0.2.1", :use_dhcp_assigned_default_route => false
streisand.vm.provision "shell", run: "always", inline: "ip route add 0.0.0.0/1 via 192.0.2.254 ; ip route add 128.0.0.0/1 via 192.0.2.254"
I’ve previously made mention of my addiction to Google Play Music… but I was called out recently, and asked about the script I used at the time. I’m sorry to say that I have had some issues with it, and instead, have resorted to using X forwarding. Here’s how I do it.
I create a user account for that other person (note, GMM will only let you upload to 3 accounts using this method. For any more, you’ll need a virtual machine!).
I then create an SSH public/private key with no passphrase.
ssh-keygen -b 2048 -N “” -C “$(whoami)@localhost” -f ~/.ssh/gmm.id_rsa
I write the public key into that new user’s .ssh/authorized_keys, by running:
ssh-copy-id -i ~/.ssh/gmm.id_rsa bloggsf@localhost
I will be prompted for the password of that account.
Finally, I create this script:
This is then added to the startup tasks of my headless-but-running-a-desktop machine.
Because of templates I was building in my post “Today I learned… Ansible Include Templates”, I thought you could repeat the same sections over again. Here’s a snippet of something like what I’d built (after combining lots of templates together):
#cloud-config
packages:
- iperf
- git
write_files:
- content: {% include 'files/public_key.j2' %}
path: /root/.ssh/authorized_keys
owner: root:root
permission: '0600'
- content: {% include 'files/private_key.j2' %}
path: /root/.ssh/id_rsa
owner: root:root
permission: '0600'
packages:
- byobu
write_files:
- content: |
#!/bin/bash
git clone {{ test_scripts }} /root/iperf_scripts
bash /root/iperf_scripts/run_test.sh
path: /root/run_test
owner: root:root
permission: '0700'
runcmd:
- /root/run_test
I’d get *bits* of it to run – basically, the last file, the last package and the last runcmd… but not all of it.
Turns out, cloud-init doesn’t like having to rebuild all the fragments together. Instead, you need to put them all together, so the write_files items, and the packages items all live in the same area.
Which, when you think about what it’s doing, which is that the parent lines are defining a variable called… well, whatever that line is, and if you replace it, it’s only going to keep the last one, then it all makes sense really!
Today I have been debugging why my Cloud-init scripts weren’t triggering on my Openstack environment.
I realised that something was wrong when I tried to use the noVNC console[1] with a password I’d set… no luck. So, next I ran a command to review the console logs[2], and saw a message (now, sadly, long gone – so I can’t even include it here!) suggesting there was an issue parsing my YAML file. Uh oh!
I’m using Ansible’s os_server module, and using templates to complete the userdata field, which in turn gets populated as cloud-init scripts…. and so clearly I had two ways to debug this – prefix my ansible playbook with a few debug commands, but then that can get messy… OR SSH into the box, and look through the logs. I knew I could SSH in, so the cloud-init had partially fired, but it just wasn’t parsing what I’d submitted. I had a quick look around, and found a post which mentioned debugging cloud-init. This mentioned that there’s a path (/var/lib/cloud/instances/$UUID/) you can mess around in, to remove some files to “fool” cloud-init into thinking it’s not been run… but, I reasoned, why not just see what’s there.
And in there, was the motherlode – user-data.txt…. bingo.
In the jinja2 template I was using to populate the userdata, I’d referenced another file, again using a template. It looks like that template needs an extra line at the end, otherwise, it all runs together.
Whew!
This does concern me a little, as I had previously been using this stanza to “simply” change the default user password to something a little less complicated:
#cloud-config
ssh_pwauth: True
chpasswd:
list: |
ubuntu:{{ default_password }}
expire: False
But now that I look at the documentation, I realise you can also specify that as a pre-hashed value (in which case, you would suffix that default_password item above with |password_hash('sha512')) which makes it all better again!
[1] If you run openstack --os-cloud cloud_a console url show servername gives you a URL to visit that has an HTML5 based VNC-ish client. Note the “cloud_a” and “servername” should be replaced by your clouds.yml reference and the server name or server ID you want to connect to.
[2] Like before, openstack --os-cloud cloud_a console log show servername gives you the output of the boot sequence (e.g. dmesg plus the normal startup commands, and finally, cloud-init). It can be useful. Equally, it’s logs… which means there’s a lot to wade through!
Thanks to my colleague Simon (@sipart on Twitter), I spotted this post (and it’s companion Github Repository) which explains how to do test-driven development in Ansible.
Essentially, you create two roles – test (the author referred to it as “validate”) and one to actually do the thing you want it to do (in the author’s case “add_vlan”).
In the testing role, you’d have the following layout:
/path/to/roles/testing/tasks/main.yml
/path/to/roles/testing/tasks/SOMEFEATUREtest.yml
In the main.yml file, you have a simple stanza:
---
- name: Include all the test files
include: "{{ outer_item }}"
with_fileglob:"/path/to/roles/validate/tasks/*test.yml"
loop_control: loop_var=outer_item
I’m sure that “with_fileglob” line could be improved to not actually need a full path… anyway
Then in your YourFeature_test.yml file, you do things like this:
---
- name: "Pseudocode in here. Use real modules for your testing!!"
get_vlan_config: filter_for=needle_vlan
register:haystack_var
- assert: that=" {{ needle_item }} in haystack_var "
When you run the play of the role the first time, the response will be “failed” (because “needle_vlan” doesn’t exist). Next do the “real” play of the role (so, in the author’s case, add_vlan) which creates the vlan. Then re-run the test role, your response should now be “ok”.
I’d probably script this so that it goes:
The benefit to doing it that way is that you “know” your tests aren’t running if the environment doesn’t have the “set_testing” thing in place, you get to run all your tests in a “clean room”, and then you clear it back down again afterwards, leaving it clear for the next pass of your automated testing suite.
Fun!
Recently, I have used K5, which is an instance of OpenStack, run by Fujitsu (my employer). To do some of the automation tasks I have played with both python-openstackclient and Ansible. This post is going to cover how to get those tools to work with K5.
I have access to a Linux virtual machine (Ubuntu 16.04) and the Windows Subsystem for Linux in Windows 10 to run “Bash on Ubuntu on Windows”, and both accept the same set of commands.
In order to run these commands, you need a couple of dependencies. Your mileage might vary with other Linux distributions, but, for Ubuntu based distributions, run this command:
sudo apt install python-pip build-essential libssl-dev libffi-dev python-dev
Next, use pip to install the python modules you need:
sudo -H pip install shade==1.11.1 ansible cryptography python-openstackclient
If you’re only ever going to be working with a single project, you can define a handful of environment variables prefixed OS_, like this:
export OS_USERNAME=BloggsF
export OS_PASSWORD=MySuperSecretPasswordIsHere
export OS_REGION_NAME=uk-1
export OS_USER_DOMAIN_NAME=YourProjectName
export OS_PROJECT_NAME=YourProjectName-prj
export OS_PROJECT_ID=baddecafbaddecafbaddecafbaddecaf
export OS_AUTH_URL=https://identity.uk-1.cloud.global.fujitsu.com/v3
export OS_VOLUME_API_VERSION=2
export OS_IDENTITY_API_VERSION=3
But, if you’re working with a few projects, it’s probably worth separating these out into clouds.yml files. This would be stored in ~/.config/openstack/clouds.yml with the credentials for the environment you’re using:
---
clouds:
root:
identity_api_version: 3
regions:
- uk-1
auth:
auth_url: https://identity.uk-1.cloud.global.fujitsu.com/v3
password: MySuperSecretPasswordIsHere
project_id: baddecafbaddecafbaddecafbaddecaf
project_name: YourProjectName-prj
username: BloggsF
user_domain_name: YourProjectName
Optionally, you can separate out the password, username or any other “sensitive” information into a secure.yml file stored in the same location (removing those lines from the clouds.yml file), like this:
---
clouds:
root:
auth:
password: MySuperSecretPasswordIsHere
Now, you can use the Python based Openstack Client, using this invocation:
openstack --os-cloud root server list
Alternatively you can use the Ansible Openstack (and K5) modules, like this:
---
tasks:
- name: "Authenticate to K5"
k5_auth:
cloud: root
register: k5_auth_reg
- name: "Create Network"
k5_create_network:
name: "Public"
availability_zone: "uk-1a"
state: present
k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Create Subnet"
k5_create_subnet:
name: "Public"
network_name: "Public"
cidr: "192.0.2.0/24"
gateway_ip: "192.0.2.1"
availability_zone: "uk-1a"
state: present
k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Create Router"
k5_create_router:
name: "Public"
availability_zone: "uk-1a"
state: present
k5_auth: "{{ k5_auth_reg.k5_auth_facts }}"
- name: "Attach private network to router"
os_router:
name: "Public"
state: present
network: "inf_az1_ext-net02"
interfaces: "Public"
cloud: root
- name: "Create Servers"
os_server:
name: "Server"
availability_zone: "uk-1a"
flavor: "P-1"
state: present
key_name: "MyFirstKey"
network: "Public-Network"
image: "Ubuntu Server 14.04 LTS (English) 02"
boot_from_volume: yes
terminate_volume: yes
security_groups: "Default"
auto_ip: no
timeout: 7200
cloud: root
This post has been revised since it was initially published on 31st March due to errors found in the resulting build. It was also missing details on the shared data drive between the two machines, so has been amended to include that.
** WARNING ** This works for me – it might not for you!
The outcome of this build will leave you with the following:
Boot up, go through the VeraCrypt bootloader, enter a password for Windows, or press escape to load the Grub bootloader where you will boot (K|L|X|)Ubuntu(| Mate| Gnome).
The Windows environment will be encrypted with VeraCrypt, an open source Full Disk Encryption technology, while the Linux environment will be encrypted using Luks. The shared volume (between Windows and Linux) will be encrypted with VeraCrypt.
PLEASE BE AWARE THAT ANY WINDOWS 10 UPGRADES WILL FAIL TO APPLY AS IT WILL NOT RECOGNISE THE VERACRYPT FILE SYSTEM! To resolve this, decrypt the Windows volume, perform the upgrade, re-encrypt it, then transfer the new recovery ISO image to the boot volume, following the method below. Yes, this will take some time. No, you don’t need to decrypt the data volume. Yes, you can use that data volume to shunt the ISO image around.
LATE EDIT 2020-01-06: I’ve just spotted a link to this post over on Level1Techs. In that post, someone asked if the broken upgrades is still a thing. Turns out that since I wrote this in 2017, it’s not been fixed. Now, I should stress, I’ve stopped using this layout as I went all-Linux on that machine, but… it might work for you now?! Also, shout out to 92aceshigh for referencing this post, and glad something I wrote helped you! ☺
My partition table, for a 320GB Disk looks (roughly) like this:
Partition 1: 20GB – Linux /Boot (ext2, plus space for ISO files for random booting)
Partition 2: 60GB – Windows C:\ (NTFS VeraCrypt)
Partition 3: 72GB – Linux Physical Volume (LVM PV, Luks Encrypted)
– logical volume 1: 16Gb Swap (Linux Swap)
– logical volume 2: 60Gb Linux (ext4)
Partition 4: 156GB – Shared Volume (NTFS, VeraCrypt)
I performed this using GParted in the Gnome Live image using the GParted. Some rational here:
The following steps need to be run as root.
sudo -i
cryptsetup luksFormat -y -v /dev/sda3
cryptsetup luksOpen /dev/sda3 lvm-pv
vgcreate vg00 /dev/mapper/lvm-pv
lvcreate -n lv00_swap -L 16G vg00 # Define 16GB Swap Space
lvcreate -n lv01_root -l +100%FREE vg00 # Define the rest of vg00 as /
LEAVE YOUR TERMINAL OPEN
Note that when you perform your install, when you get to the partitioning screen, select “Manual”, and then pick out the following volumes:
/dev/mapper/vg00-lv01_root = ext4 formatted, mount point: /
/dev/mapper/vg00-lv00_swap = swap
/dev/sda1 = ext2, format, mount point: /boot
Select the boot volume of /dev/sda. But wait, I hear you say, Windows has a well know history of nuking Grub partitions… Well, we’ll sort that in a bit…
DON’T EXIT THE LIVE SESSION ONCE THE INSTALL HAS FINISHED (select “Continue Testing”).
Go back to your terminal session. It should still be logged in as root. We need to re-mount all the partitions…
mount /dev/mapper/vg00-lv01_root /target
mount /dev/sda1 /target/boot
for i in /dev /dev/pts /proc /sys /run; do sudo mount -B $i /target$i; done
chroot /target
echo "LinuxRoot UUID=`blkid | grep sda3 | cut -d\\\" -f2` none luks" > /etc/crypttab
update-initramfs -u
grub-install --force /dev/sda1
chattr +i /boot/grub/i386-pc/core.img
update-grub
The first part installs grub to the boot position, even though it doesn’t like it, and the second forces the core file to be unchangeable… I’m not exactly sure of the impact of this, but it’s the only way to do the next part of this process. The last bit makes sure that you’ve got the latest grub config files installed.
Just check to make sure the machine boots OK!
You should have a booting Ubuntu derivative with an encrypted file system. Awesome.
Now let’s install Windows!
You should boot from your install media, when you get to the partition selector, there should only be a single NTFS partition for it to use. Use it.
Install the latest version of Veracrypt from https://veracrypt.codeplex.com/
Once it’s installed, go to System, Encrypt System Partition/Drive, “Normal” system encryption, Windows System Partition, Multi-Boot (accept the warning), Boot Drive “Yes”, Single Disk, “Non-Windows Boot Loader” – No, and then… let it go through all the rest of the steps. There will be one stage where it asks you to create a rescue disk. Just save it for later. Once the encryption settings are collected, it will do a test (which is basically just rebooting to the boot loader, having you put in your password and going back into Windows), and then let it start performing the encryption.
Once the encryption finishes, reboot the computer, enter the decryption password and test it boots to Windows OK. Then reboot it again and press escape instead of putting in the password. It will boot to your Ubuntu system.
So, there you have it. One Dual-Boot system with encrypted disks everywhere!
After you’ve got the Ubuntu and Windows volumes sorted out, next we need the shared data volume to be organised. You’ll need Veracrypt for Ubuntu. Use the following to install the Veracrypt package for Ubuntu:
sudo add-apt-repository ppa:unit193/encryption sudo apt-get update sudo apt-get install veracrypt
Once that’s installed, boot back into Windows and create a new volume – perhaps V: for Veracrypt, or E: for Encrypted – your choice, but make sure you create it using the same password that you used for the Windows partition.
Format this new volume with either NTFS or FAT32 so that you can mount it under either operating system. I chose NTFS.
Now, you need to go into Veracrypt’s Settings menu, and select “System Encryption Settings”. Tick “Cache pre-boot authentication password in driver memory” (be aware, this means that if your machine is compromised when powered up, the password could be recovered), then OK. This may prompt you to accept the UAC at this point.
Next, with the mounted volume selected, go to the “Favorites” menu, and choose “Add to System Favorites”. In the screen which comes up, select the box under “Global Settings” which says “Mount system favorite volumes when Windows starts (in the initial phase of the startup procedure)”. There will be a warning about passwords that appears. Click OK.
You may, at this point, want to move certain aspects of your Windows desktop (e.g. the “My Documents” location) to the new mounted drive.
On the Linux OS, become root, with sudo, and then add the following lines to your crontab:
@reboot mkdir -p /shared_storage @reboot veracrypt --text --non-interactive --fs-options=uid=1000,gid=1000,umask=0077 --password="YOURSUPERSECUREPASSWORD" /dev/sda4 /shared_storage
These assume that your login user’s ID is 1000 (you can check that by running the command “id” as your logged in user), that you want to use “/shared_storage” as the mount point (it stops Ubuntu treating it as a “Mountable Volume” if it’s not in your home directory and not in /mnt or /media). These options also mean that only that user (and root) can access any of the files in that partition (although, it is only you on this laptop… right?), which means you can safely use it for any files which check user permissions before allowing you to access them (e.g. SSH keys). I then set up a symbolic link to /home/MYUSERACCOUNT/Documents into the /shared_storage/Documents directory, and /home/MYUSERACCOUNT/.ssh into the /shared_storage/SSH_Keys directory.
The following list of resources helped me out when I was struggling with what to do next! They may not be canonical sources, but they helped.
I may or may not have reinstalled Windows and Kubuntu about 20 times during this process, cursing myself for starting the whole damn process off in the first place!!!
My new job means I’m currently building a lot of test boxes with Ansible, particularly OpenStack guests. This means I’m trying to script as much as possible without actually … getting my hands dirty with the actual “logging into it and running things” perspective.
This week, I hit a problem standing up a popular firewall vendor’s machine with Ansible, because I was trying to bypass the first-time-wizard… anyway, it wasn’t working, and I couldn’t figure out why. I talked to my colleague [mohclips] and he eventually told me that I needed to use a template, because what I was trying to do was too complicated.
But, damn him, I knew that wasn’t the answer :)
Anyway, I found this comment on a ticket, which lead me to the following… if you’re finding that your userdata: variable in the os_server module of Ansible isn’t working, you might need to wrap it up like this:
userdata: |
{%- raw -%}#!/bin/bash
# Kill script if the pipe fails
set -euf -o pipefail
# Write everything from this point on to Syslog
echo " == Set admin credentials == "
clish -c 'set user admin password-hash {% endraw -%}{{ default_password|password_hash('sha512') }}{%- raw -%}' -s
{% endraw %}
Note that, if you have a space before your variable, use {% endraw -%} and if you’ve a space after it, use {%- raw %} as the hyphen means “ditch all the spaces before/after this command”.