My current Ansible project relies on me collecting a lot of data from AWS and then checking it again later, to see if something has changed.
This is great for one-off tests (e.g. terraform destroy ; terraform apply ; ansible-playbook run.yml) but isn’t great for repetitive tests, especially if you have to collect data that may take many minutes to run all the actions, or if you have slow or unreliable internet in your development environment.
To get around this, I wrote a wrapper for caching this data.
At the top of my playbook, run.yml, I have these tasks:
- name: Set Online Status.
# This stores the value of run_online, unless run_online
# is not set, in which case, it defines it as "true".
ansible.builtin.set_fact:
run_online: |-
{{- run_online | default(true) | bool -}}
- name: Create cache_data path.
# This creates a "cached_data" directory in the same
# path as the playbook.
when: run_online | bool and cache_data | default(false) | bool
delegate_to: localhost
run_once: true
file:
path: "cached_data"
state: directory
mode: 0755
- name: Create cache_data for host.
# This creates a directory under "cached_data" in the same
# path as the playbook, with the name of each of the inventory
# items.
when: run_online | bool and cache_data | default(false) | bool
delegate_to: localhost
file:
path: "cached_data/{{ inventory_hostname }}"
state: directory
mode: 0755
Running this sets up an expectation for the normal operation of the playbook, that it will be “online”, by default.
Then, every time I need to call something “online”, for example, collect EC2 Instance Data (using the community.aws.ec2_instance_info module), I call out to (something like) this set of tasks, instead of just calling the task by itself.
- name: List all EC2 instances in the regions of interest.
when: run_online | bool
community.aws.ec2_instance_info:
region: "{{ item.region_name }}"
loop: "{{ regions }}"
loop_control:
label: "{{ item.region_name }}"
register: regional_ec2
- name: "NOTE: Set regional_ec2 data path"
when: not run_online | bool or cache_data | default(false) | bool
set_fact:
regional_ec2_cached_data_file_loop: "{{ regional_ec2_cached_data_file_loop | default(0) | int + 1 }}"
cached_data_filename: "cached_data/{{ inventory_hostname }}/{{ cached_data_file | default('regional_ec2') }}.{{ regional_ec2_cached_data_file_loop | default(0) | int + 1 }}.json"
- name: "NOTE: Cache/Get regional_ec2 data path"
when: not run_online | bool or cache_data | default(false) | bool
debug:
msg: "File: {{ cached_data_filename }}"
- name: Cache all EC2 instances in the regions of interest.
when: run_online | bool and cache_data | default(false) | bool
delegate_to: localhost
copy:
dest: "{{ cached_data_filename }}"
mode: "0644"
content: "{{ regional_ec2 }}"
- name: "OFFLINE: Load all EC2 instances in the regions of interest."
when: not run_online | bool
set_fact:
regional_ec2: "{% include( cached_data_filename ) %}"
The first task, if it’s still set to being “online” will execute the task, and registers the result for later. If cache_data is configured, we generate a filename for the caching, record the filename to the log (via the debug task) and then store it (using the copy task). So far, so online… but what happens when we don’t need the instance to be up and running?
In that case, we use the set_fact module, triggered by running the playbook like this: ansible-playbook run.yml -e run_online=false. This reads the cached data out of that locally stored pool of data for later use.
I recently got a new laptop, and for various reasons, I’m going to be primarily running Windows on that laptop. However, I still like having a working SSH server, running in the context of my Windows Subsystem for Linux (WSL) environment.
Initially, trying to run service ssh start failed with an error, because you need to re-execute the ssh configuration steps which are missed in a WSL environment. To fix that, run sudo apt install --reinstall openssh-server.
Once you know your service runs OK, you start digging around to find out how to start it on boot, and you’ll see lots of people saying things like “Just run a shell script that starts your first service, and then another shell script for the next service.”
Well, the frustration for me is that Linux already has this capability – the current popular version is called SystemD, but a slightly older variant is still knocking around in modern linux distributions, and it’s called SystemV Init, often referred to as just “sysv” or “init.d”.
The way that those services work is that you have an “init” file in /etc/init.d and then those files have a symbolic link into a “runlevel” directory, for example /etc/rc3.d. Each symbolic link is named S##service or K##service, where the ## represents the order in which it’s to be launched. The SSH Daemon, for example, that I want to run is created in there as /etc/rc3.d/S01ssh.
So, how do I make this work in the grander scheme of WSL? I can’t use SystemD, where I could say systemctl enable --now ssh, instead I need to add a (yes, I know) shell script, which looks in my desired runlevel directory. Runlevel 3 is the level at which network services have started, hence using that one. If I was trying to set up a graphical desktop, I’d instead be looking to use Runlevel 5, but the X Windows system isn’t ported to Windows like that yet… Anyway.
Because the rc#.d directory already has this structure for ordering and naming services to load, I can just step over this directory looking for files which match or do not match the naming convention, and I do that with this script:
#! /bin/bash
function run_rc() {
base="$(basename "$1")"
if [[ ${base:0:1} == "S" ]]
then
"$1" start
else
"$1" stop
fi
}
if [ "$1" != "" ] && [ -e "$1" ]
then
run_rc "$1"
else
rc=3
if [ "$1" != "" ] && [ -e "/etc/rc${$1}.d/" ]
then
rc="$1"
fi
for digit1 in {0..9}
do
for digit2 in {0..9}
do
find "/etc/rc${rc}.d/" -name "[SK]${digit1}${digit2}*" -exec "$0" '{}' \; 2>/dev/null
done
done
fi
I’ve put this script in /opt/wsl_init.sh
This does a bit of trickery, but basically runs the bottom block first. It loops over the digits 0 to 9 twice (giving you 00, 01, 02 and so on up to 99) and looks in /etc/rc3.d for any file containing the filename starting S or K and then with the two digits you’ve looped to by that point. Finally, it runs itself again, passing the name of the file it just found, and this is where the top block comes in.
In the top block we look at the “basename” – the part of the path supplied, without any prefixed directories attached, and then extract just the first character (that’s the ${base:0:1} part) to see whether it’s an “S” or anything else. If it’s an S (which everything there is likely to be), it executes the task like this: /etc/rc3.d/S01ssh start and this works because it’s how that script is designed! You can run one of the following instances of this command: service ssh start, /etc/init.d/ssh start or /etc/rc3.d/S01ssh start. There are other options, notably “stop” or “status”, but these aren’t really useful here.
Now, how do we make Windows execute this on boot? I’m using NSSM, the “Non-sucking service manager” to add a line to the Windows System services. I placed the NSSM executable in C:\Program Files\nssm\nssm.exe, and then from a command line, ran C:\Program Files\nssm\nssm.exe install WSL_Init.
I configured it with the Application Path: C:\Windows\System32\wsl.exe and the Arguments: -d ubuntu -e sudo /opt/wsl_init.sh. Note that this only works because I’ve also got Sudo setup to execute this command without prompting for a password.
Here I invoke C:\Windows\System32\wsl.exe -d ubuntu -e sudo /opt/wsl_init.shI define the name of the service, as Services will see it, and also the description of the service.I put in MY username and My Windows Password here, otherwise I’m not running WSL in my user context, but another one.
And then I rebooted. SSH was running as I needed it.
For a project I’m working on, I needed to define a list of ports, and set some properties on some of them. In the Ansible world, you’d use statements like:
{% if data.somekey is defined %}something {{ data.somekey }}{% endif %}
or
{{ data.somekey | default('') }}
In a pinch, you can also do this:
{{ (data | default({}) ).somekey | default('') }}
With Terraform, I was finding it much harder to work out how to find whether a value as part of a map (the Terraform term for a Dictionary in Ansible terms, or an Associative Array in PHP terms), until I stumbled over the Lookup function. Here’s how that looks for just a simple Terraform file:
In a project I’m working on in Terraform, I’ve got several feature flags in a module. These flags relate to whether this module should turn on a system in a cloud provider, or not, and looks like this:
variable "turn_on_feature_x" {
description = "Setting this to 'yes' will enable Feature X. Any other value will disable it. (Default 'yes')"
value = "yes"
}
variable "turn_on_feature_y" {
description = "Setting this to 'yes' will enable Feature Y. Any other value will disable it. (Default 'no')"
value = "no"
}
When I call the module, I then can either leave the feature with the default values, or selectively enable or disable them, like this:
When I then want to use the feature, I have to remember a couple of key parts.
Normally this feature check is done with a “count” statement, and the easiest way to use this is to use the ternary operator to check values and return a “1” or a “0” for if you want the value used.
Ternary operators look like this: var.turn_on_feature_x == "yes" ? 1 : 0 which basically means, if the value of the variable turn_on_feature_x is set to “yes”, then return 1 otherwise return 0.
This can get a bit complex, particularly if you want to check several flags a few times, like this: var.turn_on_feature_x == "yes" ? var.turn_on_feature_y == "yes" ? 1 : 0 : 0. I’ve found that wrapping them in brackets helps to understand what you’re getting, like this:
If you end up using a count statement, the resulting value must be treated as an 0-indexed array, like this: some_provider_service.my_name[0].result
This is because, using the count value says “I want X number of resources”, so Terraform has to treat it as an array, in case you actually wanted 10 instead of 1 or 0.
TL;DR?It’s possible to work out what type of variable you’re working with in Ansible. The in-built filters don’t always do quite what you’re expecting. Jump to the “In Summary” heading for my suggestions.
One of the things I end up doing quite a bit with Ansible is value manipulation. I know it’s not really normal, but… well, I like rewriting values from one type of a thing to the next type of a thing.
For example, I like taking a value that I don’t know if it’s a list or a string, and passing that to an argument that expects a list.
Doing it wrong, getting it better
Until recently, I’d do that like this:
- debug:
msg: |-
{
{%- if value | type_debug == "string" or value | type_debug == "AnsibleUnicode" -%}
"string": "{{ value }}"
{%- elif value | type_debug == "dict" or value | type_debug == "ansible_mapping" -%}
"dict": {{ value }}
{%- elif value | type_debug == "list" -%}
"list": {{ value }}
{%- else -%}
"other": "{{ value }}"
{%- endif -%}
}
But, following finding this gist, I now know I can do this:
- debug:
msg: |-
{
{%- if value is string -%}
"string": "{{ value }}"
{%- elif value is mapping -%}
"dict": {{ value }}
{%- elif value is iterable -%}
"list": {{ value }}
{%- else -%}
"other": "{{ value }}"
{%- endif -%}
}
So, how would I use this, given the context of what I was saying before?
- assert:
that:
- value is string
- value is not mapping
- value is iterable
- some_module:
some_arg: |-
{%- if value is string -%}
["{{ value }}"]
{%- else -%}
{{ value }}
{%- endif -%}
More details on finding a type
Why in this order? Well, because of how values are stored in Ansible, the following states are true:
⬇️Type \ ➡️Check
is iterable
is mapping
is sequence
is string
a_dict (e.g. {})
✔️
✔️
✔️
❌
a_list (e.g. [])
✔️
❌
✔️
❌
a_string (e.g. “”)
✔️
✔️
✔️
✔️
A comparison between value types
So, if you were to check for is iterable first, you might match on a_list or a_dict instead of a_string, but string can only match on a_string. Once you know it can’t be a string, you can check whether something is mapping – again, because a mapping can match either a_string or a_dict, but it can’t match a_list. Once you know it’s not that, you can check for either is iterable or is sequence because both of these match a_string, a_dict and a_list.
Likewise, if you wanted to check whether a_float and an_integeris number and not is string, you can check these:
⬇️Type \ ➡️Check
is float
is integer
is iterable
is mapping
is number
is sequence
is string
a_float
✔️
❌
❌
❌
✔️
❌
❌
an_integer
❌
✔️
❌
❌
✔️
❌
❌
A comparison between types of numbers
So again, a_float and an_integer don’t match is string, is mapping or is iterable, but they both match is number and they each match their respective is float and is integer checks.
How about each of those (a_float and an_integer) wrapped in quotes, making them a string? What happens then?
⬇️Type \ ➡️Check
is float
is integer
is iterable
is mapping
is number
is sequence
is string
a_float_as_string
❌
❌
✔️
❌
❌
✔️
✔️
an_integer_as_string
❌
❌
✔️
❌
❌
✔️
✔️
A comparison between types of numbers when held as a string
This is somewhat interesting, because they look like a number, but they’re actually “just” a string. So, now you need to do some comparisons to make them look like numbers again to check if they’re numbers.
Changing the type of a string
What happens if you cast the values? Casting means to convert from one type of value (e.g. string) into another (e.g. float) and to do that, Ansible has three filters we can use, float, int and string. You can’t cast to a dict or a list, but you can use dict2items and items2dict (more on those later). So let’s start with casting our group of a_ and an_ items from above. Here’s a list of values I want to use:
With each of these values, I returned the value as Ansible knows it, what happens when you do {{ value | float }} to cast it as a float, as an integer by doing {{ value | int }} and as a string {{ value | string }}. Some of these results are interesting. Note that where you see u'some value' means that Python converted that string to a Unicode string.
⬇️Value \ ➡️Cast
value
value when cast as float
value when cast as integer
value when cast as string
a_dict
{“key1”: “value1”}
0.0
0
“{u’key1′: u’value1′}”
a_float
1.1
1.1
1
“1.1”
a_float_as_string
“1.1”
1.1
1
“1.1”
a_list
[“item1”]
0.0
0
“[u’item1′]”
a_string
“string”
0.0
0
“string”
an_int
1
1
1
“1”
an_int_as_string
“1”
1
1
“1”
Casting between value types
So, what does this mean for us? Well, not a great deal, aside from to note that you can “force” a number to be a string, or a string which is “just” a number wrapped in quotes can be forced into being a number again.
Oh, and casting dicts to lists and back again? This one is actually pretty clearly documented in the current set of documentation (as at 2.9 at least!)
Checking for miscast values
How about if I want to know whether a value I think might be a float stored as a string, how can I check that?
What is this? If I cast a value that I think might be a float, to a float, and then turn both the cast value and the original into a string, do they match? If I’ve got a string or an integer, then I’ll get a false, but if I have actually got a float, then I’ll get true. Likewise for casting an integer. Let’s see what that table looks like:
⬇️Type \ ➡️Check
value when cast as float
value when cast as integer
value when cast as string
a_float
✔️
❌
✔️
a_float_as_string
✔️
❌
✔️
an_integer
❌
✔️
✔️
an_integer_as_string
❌
✔️
✔️
A comparison between types of numbers when cast to a string
So this shows us the values we were after – even if you’ve got a float (or an integer) stored as a string, by doing some careful casting, you can confirm they’re of the type you wanted… and then you can pass them through the right filter to use them in your playbooks!
Booleans
Last thing to check – boolean values – “True” or “False“. There’s a bit of confusion here, as a “boolean” can be: true or false, yes or no, 1 or 0, however, is true and True and TRUE the same? How about false, False and FALSE? Let’s take a look!
⬇️Value \ ➡️Check
type_debug
is boolean
is number
is iterable
is mapping
is string
value when cast as bool
value when cast as string
value when cast as integer
yes
bool
✔️
✔️
❌
❌
❌
True
True
1
Yes
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
Yes
0
YES
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
YES
0
“yes”
AnsibleUnicode
❌
❌
✔️
❌
✔️
True
yes
0
“Yes”
AnsibleUnicode
❌
❌
✔️
❌
✔️
True
Yes
0
“YES”
AnsibleUnicode
❌
❌
✔️
❌
✔️
True
YES
0
true
bool
✔️
✔️
❌
❌
❌
True
True
1
True
bool
✔️
✔️
❌
❌
❌
True
True
1
TRUE
bool
✔️
✔️
❌
❌
❌
True
True
1
“true”
AnsibleUnicode
❌
❌
✔️
❌
✔️
True
true
0
“True”
AnsibleUnicode
❌
❌
✔️
❌
✔️
True
True
0
“TRUE”
AnsibleUnicode
❌
❌
✔️
❌
✔️
True
TRUE
0
1
int
❌
✔️
❌
❌
❌
True
1
1
“1”
AnsibleUnicode
❌
❌
✔️
❌
✔️
True
1
1
no
bool
✔️
✔️
❌
❌
❌
False
False
0
No
bool
✔️
✔️
❌
❌
❌
False
False
0
NO
bool
✔️
✔️
❌
❌
❌
False
False
0
“no”
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
no
0
“No”
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
No
0
“NO”
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
NO
0
false
bool
✔️
✔️
❌
❌
❌
False
False
0
False
bool
✔️
✔️
❌
❌
❌
False
False
0
FALSE
bool
✔️
✔️
❌
❌
❌
False
False
0
“false”
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
false
0
“False”
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
False
0
“FALSE”
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
FALSE
0
0
int
❌
✔️
❌
❌
❌
False
0
0
“0”
AnsibleUnicode
❌
❌
✔️
❌
✔️
False
0
0
Comparisons between various stylings of boolean representations
So, the stand out thing for me here is that while all the permutations of string values of the boolean representations (those wrapped in quotes, like this: "yes") are treated as strings, and shouldn’t be considered as “boolean” (unless you cast for it explicitly!), and all non-string versions of true, false, and no are considered to be boolean, yes, Yes and YES are treated differently, depending on case. So, what would I do?
In summary
Consistently use no or yes, true or false in lower case to indicate a boolean value. Don’t use 1 or 0 unless you have to.
If you’re checking that you’re working with a string, a list or a dict, check in the order string (using is string), dict (using is mapping) and then list (using is sequence or is iterable)
Checking for numbers that are stored as strings? Cast your string through the type check for that number, like this: {% if value | float | string == value | string %}{{ value | float }}{% elif value | int | string == value | string %}{{ value | int }}{% else %}{{ value }}{% endif %}
Try not to use type_debug unless you really can’t find any other way. These values will change between versions, and this caused me a lot of issues with a large codebase I was working on a while ago!
Run these tests yourself!
Want to run these tests yourself? Here’s the code I ran (also available in a Gist on GitHub), using Ansible 2.9.10.
---
- hosts: localhost
gather_facts: no
vars:
an_int: 1
a_float: 1.1
a_string: "string"
an_int_as_string: "1"
a_float_as_string: "1.1"
a_list:
- item1
a_dict:
key1: value1
tasks:
- debug:
msg: |
{
{% for var in ["an_int", "an_int_as_string","a_float", "a_float_as_string","a_string","a_list","a_dict"] %}
"{{ var }}": {
"type_debug": "{{ vars[var] | type_debug }}",
"value": "{{ vars[var] }}",
"is float": "{{ vars[var] is float }}",
"is integer": "{{ vars[var] is integer }}",
"is iterable": "{{ vars[var] is iterable }}",
"is mapping": "{{ vars[var] is mapping }}",
"is number": "{{ vars[var] is number }}",
"is sequence": "{{ vars[var] is sequence }}",
"is string": "{{ vars[var] is string }}",
"value cast as float": "{{ vars[var] | float }}",
"value cast as integer": "{{ vars[var] | int }}",
"value cast as string": "{{ vars[var] | string }}",
"is same when cast to float": "{{ vars[var] | float | string == vars[var] | string }}",
"is same when cast to integer": "{{ vars[var] | int | string == vars[var] | string }}",
"is same when cast to string": "{{ vars[var] | string == vars[var] | string }}",
},
{% endfor %}
}
---
- hosts: localhost
gather_facts: false
vars:
# true, True, TRUE, "true", "True", "TRUE"
a_true: true
a_true_initial_caps: True
a_true_caps: TRUE
a_string_true: "true"
a_string_true_initial_caps: "True"
a_string_true_caps: "TRUE"
# yes, Yes, YES, "yes", "Yes", "YES"
a_yes: yes
a_yes_initial_caps: Tes
a_yes_caps: TES
a_string_yes: "yes"
a_string_yes_initial_caps: "Yes"
a_string_yes_caps: "Yes"
# 1, "1"
a_1: 1
a_string_1: "1"
# false, False, FALSE, "false", "False", "FALSE"
a_false: false
a_false_initial_caps: False
a_false_caps: FALSE
a_string_false: "false"
a_string_false_initial_caps: "False"
a_string_false_caps: "FALSE"
# no, No, NO, "no", "No", "NO"
a_no: no
a_no_initial_caps: No
a_no_caps: NO
a_string_no: "no"
a_string_no_initial_caps: "No"
a_string_no_caps: "NO"
# 0, "0"
a_0: 0
a_string_0: "0"
tasks:
- debug:
msg: |
{
{% for var in ["a_true","a_true_initial_caps","a_true_caps","a_string_true","a_string_true_initial_caps","a_string_true_caps","a_yes","a_yes_initial_caps","a_yes_caps","a_string_yes","a_string_yes_initial_caps","a_string_yes_caps","a_1","a_string_1","a_false","a_false_initial_caps","a_false_caps","a_string_false","a_string_false_initial_caps","a_string_false_caps","a_no","a_no_initial_caps","a_no_caps","a_string_no","a_string_no_initial_caps","a_string_no_caps","a_0","a_string_0"] %}
"{{ var }}": {
"type_debug": "{{ vars[var] | type_debug }}",
"value": "{{ vars[var] }}",
"is float": "{{ vars[var] is float }}",
"is integer": "{{ vars[var] is integer }}",
"is iterable": "{{ vars[var] is iterable }}",
"is mapping": "{{ vars[var] is mapping }}",
"is number": "{{ vars[var] is number }}",
"is sequence": "{{ vars[var] is sequence }}",
"is string": "{{ vars[var] is string }}",
"is bool": "{{ vars[var] is boolean }}",
"value cast as float": "{{ vars[var] | float }}",
"value cast as integer": "{{ vars[var] | int }}",
"value cast as string": "{{ vars[var] | string }}",
"value cast as bool": "{{ vars[var] | bool }}",
"is same when cast to float": "{{ vars[var] | float | string == vars[var] | string }}",
"is same when cast to integer": "{{ vars[var] | int | string == vars[var] | string }}",
"is same when cast to string": "{{ vars[var] | string == vars[var] | string }}",
"is same when cast to bool": "{{ vars[var] | bool | string == vars[var] | string }}",
},
{% endfor %}
}
Featured image is “Kelvin Test” by “Eelke” on Flickr and is released under a CC-BY license.
In it, Ricardo introduces me to two things which are interesting.
Using the wait command literally waits for all the backgrounded tasks to finish.
Running bash commands like this: function1 & function2 & function3 should run all three processes in parallel. To be honest, I’d always usually do it like this: function1 & function2 & function3 &
The other thing which Ricardo links to is a page suggesting that if you’re downloading a bash script and executing it (which, you know, probably isn’t a good idea at the best of times), then wrapping it in a function, like this:
#!/bin/bash
function main() {
echo "Some function"
}
main
This means that the bash scripting engine needs to download and parse all the functions before it can run the script. As a result, you’re less likely to get a broken run of your script, because imagine it only got as far as:
#!/bin/bash
echo "Some fun
Then it wouldn’t have terminated the echo command (as an example)…
I’m currently building a Proof Of Value (POV) environment for a product, and one of the things I needed in my environment was an Active Directory domain.
To do this in AWS, I had to do the following steps:
Build my Domain Controller
Install Windows
Set the hostname (Reboot)
Promote the machine to being a Domain Controller (Reboot)
Create a domain user
Build my Member Server
Install Windows
Set the hostname (Reboot)
Set the DNS client to point to the Domain Controller
Join the server to the domain (Reboot)
To make this work, I had to find a way to trigger build steps after each reboot. I was working with Windows 2012R2, Windows 2016 and Windows 2019, so the solution had to be cross-version. Fortunately I found this script online! That version was great for Windows 2012R2, but didn’t cover Windows 2016 or later… So let’s break down what I’ve done!
In your userdata field, you need to have two sets of XML strings, as follows:
The first block says to Windows 2016+ “keep trying to run this script on each boot” (note that you need to stop it from doing non-relevant stuff on each boot – we’ll get to that in a second!), and the second bit is the PowerShell commands you want it to run. The rest of this now will focus just on the PowerShell block.
Whew, what a block! Well, again, we can split this up into a couple of bits.
In the first few lines, we build a pointer, a note which says “We got up to here on our previous boots”. We then read that into a variable and find that number and execute any steps in the block with that number. That’s this block:
The next part (and you’ll repeat it for each “number” of reboot steps you need to perform) says “increment the number” then “If this is Windows 2012, remind the userdata handler that the script needs to be run again next boot”. That’s this block:
Then, after each reboot, you need a new block. I have a block to change the computer name, a block to join the machine to the domain, and a block to install an software that I need.
This one is more a nudge to myself. On several occasions when building Infrastructure As Code (IAC), I split out a code sections into one or more files, for readability and reusability purposes. What I tended to do, and this was more apparent with the Linux builds than the Windows builds, was to forget to set the line terminator from CRLF to LF.
While this doesn’t really impact Windows builds too much (they’re kinda designed to support people being idiots with line endings now), Linux still really struggles with CRLF endings, and you’ll only see when you’ve broken this because you’ll completely fail to run any of the user-data script.
How do you determine this is your problem? Well, actually it’s a bit tricky, as neither cat, less, more or nano spot this issue. The only two things I found that identified it were file and vi.
The first part of the combined file with mixed line endings. This part has LF termination.The second part of the combined file with mixed line endings. This part has CRLF termination.What happens when we cat these two parts into one file? A file with CRLF, LF line terminators obviously!What the combined file looks like in Vi. Note the blue ^M at the ends of the lines.
So, how to fix this? Assuming you’re using Visual Studio Code;
A failed line-ending clue in Visual Studio Code
You’ll notice this line showing “CRLF” in the status bar at the bottom of Code. Click on that, which brings up a discrete box near the top, as follows:
Oh no, it’s set to “CRLF”. That’s not what we want!
Selecting LF in that box changes the line feeds into LF for this file, but it’s not saved. Make sure you save this file before you re-run your terraform script!
Notice, we’re now using LF endings, but the file isn’t saved.
Fantastic! It’s all worked!
In Nano, I’ve opened the part with the invalid line endings.
Oh no! We have a “DOS Format” file. Quick, let’s fix it!
To fix this, we need to write the file out. Hit Ctrl+O. This tells us that we’re in DOS Format, and also gives us the keyboard combination to toggle “DOS Format” off – it’s Alt+D (In Unix/Linux world, the Alt key is referred to as the Meta key – hence M not A).
This is how we fix things
So, after hitting Alt+D, the “File Name to write” line changes, see below:
Yey, no pesky “DOS Format” warning here!
Using either editor (or any others, if you know how to solve line ending issues in other editors), you still need to combine your script back together before you can run it, so… do that, and your file will be fine to run! Good luck!
I started thinking a couple of weeks ago, when my coding laptop broke, that it would be really useful to have a development machine somewhere else that I could use.
It wouldn’t need a lot of power (after all, I’m mostly developing web apps and not compiling stuff), but it does need to be a desktop OS, as I rather like being able to open code editors and suchlike, while I’ve got a web browser open.
I have an Android tablet, which while it’s great for being a tablet, it’s not much use as a desktop, and … yes, I’ve got a work laptop, but I don’t really want to install software on that (and I don’t think my admin team would be happy if I did).
Also, I quite like Linux.
Some time ago, I spotted that AWS has a “Virtual Desktop” environment, and I think that’s kinda what I’m after. Something I can spin up, run for a bit and then shut it down, so I thought I’d build something like that… but not pesky Windows, after all… who likes Windows, eh? ;)
So, I built a Virtual Desktop Environment (VDE) in AWS, using Terraform and a bit of shell script!
I start from an Ubuntu 18.04 server image, and, after the install is complete, I run this user-data script inside it. Yes, I know I could be doing this with Ansible, but… eh, I wanted it to be a quick deployment ;)
Oh, and there’s a couple of Terraform managed variables in here – ${aws_eip.vde.public_ip} is the AWS public IP address assigned to this host., ${var.firstuser} is the username we want to rename “ubuntu” (the stock server username) to. ${var.firstgecos} is the user’s “real name” which the machine identifies the user as (like “Log out Jon Spriggs” and so on). ${var.userpw} is either the password you want it to use, OR (by default) pwgen 12 which generates a 12 character long password. ${var.desktopenv} is the name of the desktop environment I want to install (Ubuntu by default) and … well, ${var.var_start} is a bit of a fudge, because I couldn’t, in a hurry, work out how to tell Terraform not to mangle the bash variable allocation of ${somevar} which is the format that Terraform also uses. D’oh.
#! /bin/bash
#################
# Set Hostname
#################
hostnamectl set-hostname vde.${aws_eip.vde.public_ip}.nip.io
#################
# Change User
#################
user=${var.firstuser}
if [ ! "$user" == 'ubuntu' ]
then
until usermod -c "${var.firstgecos}" -l $user ubuntu ; do sleep 5 ; done
until groupmod -n $user ubuntu ; do sleep 5 ; done
until usermod -d /home/$user -m $user ; do sleep 5 ; done
if [ -f /etc/sudoers.d/90-cloudimg-ubuntu ]; then
mv /etc/sudoers.d/90-cloudimg-ubuntu /etc/sudoers.d/90-cloud-init-users
fi
perl -pi -e "s/ubuntu/$user/g;" /etc/sudoers.d/90-cloud-init-users
fi
if [ '${var.userpw}' == '$(pwgen 12)' ]
then
apt update && apt install pwgen
fi
newpw="${var.userpw}"
echo "$newpw" > /var/log/userpw
fullpw="$newpw"
fullpw+="\n"
fullpw+="$newpw"
echo -e "$fullpw" | passwd $user
##########################
# Install Desktop and RDP
##########################
apt-get update
export DEBIAN_FRONTEND=noninteractive
apt-get full-upgrade -yq
apt-get autoremove -y
apt-get autoclean -y
apt-get install -y ${var.desktopenv}-desktop xrdp certbot
##########################
# Configure Certbot
##########################
echo "#!/bin/sh" > /etc/letsencrypt/merge_cert.sh
echo 'cat ${var.var_start}{RENEWED_LINEAGE}/privkey.pem ${var.var_start}{RENEWED_LINEAGE}/fullchain.pem > ${var.var_start}{RENEWED_LINEAGE}/merged.pem' >> /etc/letsencrypt/merge_cert.sh
echo 'chmod 640 ${var.var_start}{RENEWED_LINEAGE}/merged.pem' >> /etc/letsencrypt/merge_cert.sh
chmod 750 /etc/letsencrypt/merge_cert.sh
certbot certonly --standalone --deploy-hook /etc/letsencrypt/merge_cert.sh -n -d vde.${aws_eip.vde.public_ip}.nip.io -d ${aws_eip.vde.public_ip}.nip.io --register-unsafely-without-email --agree-tos
# Based on https://www.snel.com/support/xrdp-with-lets-encrypt-on-ubuntu-18-04/
sed -i 's~^certificate=$~certificate=/etc/letsencrypt/live/vde.${aws_eip.vde.public_ip}.nip.io/fullchain.pem~; s~^key_file=$~key_file=/etc/letsencrypt/live/vde.${aws_eip.vde.public_ip}.nip.io/privkey.pem' /etc/xrdp/xrdp.ini
##############################
# Fix colord remote user issue
##############################
# Derived from http://c-nergy.be/blog/?p=12043
echo "[Allow Colord all Users]
Identity=unix-user:*
Action=org.freedesktop.color-manager.create-device;org.freedesktop.color-manager.create-profile;org.freedesktop.color-manager.delete-device;org.freedesktop.color-manager.delete-profile;org.freedesktop.color-manager.modify-device;org.freedesktop.color-manager.modify-profile
ResultAny=no
ResultInactive=no
ResultActive=yes" > /etc/polkit-1/localauthority/50-local.d/45-allow.colord.pkla
##############################
# Configure Desktop
##############################
if [ '${var.desktopenv}' == 'ubuntu' ]
then
echo "#!/bin/bash" > /tmp/desktop_settings
echo "gsettings set org.gnome.desktop.input-sources sources \"[('xkb', 'gb')]\"" >> /tmp/desktop_settings
echo "gsettings set org.gnome.desktop.app-folders folder-children \"['Utilities', 'Sundry', 'YaST']\"" >> /tmp/desktop_settings
echo "gsettings set org.gnome.desktop.privacy report-technical-problems false" >> /tmp/desktop_settings
echo "gsettings set org.gnome.desktop.screensaver lock-enabled false" >> /tmp/desktop_settings
echo "gsettings set org.gnome.desktop.session idle-delay 0" >> /tmp/desktop_settings
echo "echo yes > /home/${var.firstuser}/.config/gnome-initial-setup-done" >> /tmp/desktop_settings
sudo -H -u ${var.firstuser} dbus-launch --exit-with-session bash /tmp/desktop_settings
rm -f /tmp/desktop_settings
fi
##########################
# Install VSCode
##########################
wget https://vscode-update.azurewebsites.net/latest/linux-deb-x64/stable -O /tmp/vscode.deb
apt install -y /tmp/vscode.deb
rm /var/crash/*
shutdown -r now
Ubuntu 18.04 has a “first login” wizard, that lets you pre-set up things like, what language will you be using. I bypassed this with the gsettings commands towards the end of the script, and writing the string “yes” to ~/.config/gnome-initial-setup-done.
Also, I wanted to be able to RDP to it. I’m a bit concerned by the use of VNC, especially where RDP is more than capable. It’s just an apt-install away, so… that’s what I do. But, because I’m RDP’ing into this box, I wanted to prevent the RDP session from locking, so I provide two commands to the session: gsettings set org.gnome.desktop.screensaver lock-enabled false which removes the screensaver’s ability to lock the screen, and gsettings set org.gnome.desktop.session idle-delay 0 which stops the screensaver from even starting in the first place.
Now all I need to do is to figure out where I’m going to store my code between boots ;)
So, in summary, I now have a Virtual Machine, which runs Ubuntu 18.04 Desktop, in AWS, with an RDP connection (powered by xRDP), and a disabled screensaver. Job done, I think!
Oh, and if I’m doing it “wrong”, let me know in the comments? :)
Late edit 2020-01-16: The fantastic Jerry Steel, my co-host on The Admin Admin podcast looked at what I wrote, and made a few suggestions. I’ve updated the code in the git repo, and I’ll try to annotate below when I’ve changed something. If I miss it, it’s right in the Git repo!
One of the challenges I set myself this Christmas was to learn enough about Docker to put an arbitrary PHP application, that I would previously have misused Vagrant to contain.
Just before I started down this rabbit hole, I spoke to my Aunt about some family tree research my father had left behind after he died, and how I wished I could easily share the old tree with her (I organised getting her a Chromebook a couple of years ago, after fighting with doing remote support for years on Linux and Windows laptops). In the end, I found a web application for genealogical research called HuMo-gen, that is a perfect match for both projects I wanted to look at.
HuMo-gen was first created in 1999, with a PHP version being released in 2005. It used MySQL or MariaDB as the Database engine. I was reasonably confident that I could have created a Vagrantfile to deliver this on my home server, but I wanted to try something new. I wanted to use the “standard” building blocks of Docker and Docker-Compose, and some common containers to make my way around learning Docker.
I started by looking for some resources on how to build a Docker container. Much of the guidance I’d found was to use Docker-Compose, as this allows you to stand several components up at the same time!
In contrast to how Vagrant works (which is basically a CLI wrapper to many virtual machine services), Docker isolates resources for a single process that runs on a machine. Where in Vagrant, you might run several processes on one machine (perhaps, in this instance, nginx, PHP-FPM and MariaDB), with Docker, you’re encouraged to run each “service” as their own containers, and link them together with an overlay network. It’s possible to also do the same with Vagrant, but you’ll end up with an awful lot of VM overhead to separate out each piece.
So, I first needed to select my services. My initial line-up was:
MariaDB
PHP-FPM
Apache’s httpd2 (replaced by nginx)
I was able to find official Docker images for PHP, MariaDB and httpd, but after extensive tweaking, I couldn’t make the httpd image talk the way I wanted it to with the PHP image. Bowing to what now seems to be conventional wisdom, I swapped out the httpd service for nginx.
One of the stumbling blocks for me, particularly early on, was how to build several different Dockerfiles (these are basically the instructions for the container you’re constructing). Here is the basic outline of how to do this:
In this docker-compose.yml file, I tell it that to create the yourservice service, it needs to build the docker container, using the file in ./relative/path/to/Dockerfile. This file in turn contains an instruction to import an image.
Each service stacks on top of each other in that docker-compose.yml file, like this:
Late edit 2020-01-16: This previously listed Dockerfile/service1, however, much of the documentation suggested that Docker gets quite opinionated about the file being called Dockerfile. While docker-compose can work around this, it’s better to stick to tradition :) The docker-compose.yml files below have also been adjusted accordingly. I’ve also added an image: somehost:1234/image_name line to help with tagging the images for later use. It’s not critical to what’s going on here, but I found it useful with some later projects.
To allow containers to see ports between themselves, you add the expose: command in your docker-compose.yml, and to allow that port to be visible from the “outside” (i.e. to the host and upwards), use the ports: command listing the “host” port (the one on the host OS), then a colon and then the “target” port (the one in the container), like these:
Now, let’s take a quick look into the Dockerfiles. Each “statement” in a Dockerfile adds a new “layer” to the image. For local operations, this probably isn’t a problem, but when you’re storing these images on a hosted provider, you want to keep these images as small as possible.
I built a Database Dockerfile, which is about as small as you can make it!
FROM mariadb:10.4.10
Yep, one line. How cool is that? In the docker-compose.yml file, I invoke this, like this:
OK, so this one is a bit more complex! I wanted it to build my Dockerfile, which is “mariadb/Dockerfile“. I wanted it to restart the container whenever it failed (which hopefully isn’t that often!), and I wanted to inject some specific environment variables into the file – the root and user passwords, a user account and a database name. Initially I was having some issues where it wasn’t building the database with these credentials, but I think that’s because I wasn’t “building” the new database, I was just using it. I also expose the MariaDB (MySQL) port, 3306 to the other containers in the docker-compose.yml file.
Let’s take a look at the next part! PHP-FPM. Here’s the Dockerfile:
FROM php:7.4-fpm
RUN docker-php-ext-install pdo pdo_mysql
ADD --chown=www-data:www-data public /var/www/html
There’s a bit more to this, but not loads. We build our image from a named version of PHP, and install two extensions to PHP, pdo and pdo_mysql. Lastly, we copy the content of the “public” directory into the /var/www/html path, and make sure it “belongs” to the right user (www-data).
I’d previously tried to do a lot more complicated things with this Dockerfile, but it wasn’t working, so instead I slimmed it right down to just this, and the docker-compose.yml is a lot simpler too.
See! Loads simpler! Now we need the complicated bit! :) This is the Dockerfile for nginx.
FROM nginx:1.17.7
COPY nginx/default.conf /etc/nginx/conf.d/default.conf
COPY public /var/www/html
Weirdly, even though I’ve added version numbers for MariaDB and PHP, I’ve not done the same for nginx, perhaps I should!Late edit 2020-01-16: I’ve put a version number on there now, previously where it said nginx:1.17.7 it actually said nginx:latest.
I’ve created the configuration block for nginx in a single “RUN” line.Late edit 2020-01-16: This Dockerfile now doesn’t have a giant echo 'stuff' > file block either, following Jerry’s advice, and I’m using COPY instead of ADD on his advice too. I’ll show that config file below. There’s a couple of high points for me here!
server_name _;means “use this block for all unnamed requests”.
access_log /proc/self/fd/1; and error_log /proc/self/fd/2;These are links to the “stdout” and “stderr” file descriptors (or pointers to other parts of the filesystem), and basically means that when you do docker-compose logs, you’ll see the HTTP logs for the server! These two files are guaranteed to be there, while /dev/stderr isn’t!
Because nginx is “just” caching the web content, and I know the content doesn’t need to be written to from nginx, I knew I didn’t need to do the chown action, like I did with the PHP-FPM block.
Lastly, I need to configure the docker-compose.yml file for nginx:
I’ve gone for a slightly unusual ports configuration when I deployed this to my web server… you see, I already have the HTTP port (TCP/80) configured for use on my home server – for running the rest of my web services. During development, on my home machine, the ports line instead showed “1980:80” because I was running this on Instead, I’m running this application bound to “localhost” (127.0.0.1) on a different port number (1980 selected because it could, conceivably, be a birthday of someone on this system), and then in my local web server configuration, I’m proxying connections to this service, with HTTPS encryption as well. That’s all outside the scope of this article (as I probably should be using something like Traefik, anyway) but it shows you how you could bind to a separate port too.
Anyway, that was my Docker journey over Christmas, and I look forward to using it more, going forward!