Post-Config of a RaspberryPi Zero W as an OTG-USB Gadget for off-device computing

History

A few months ago, I was working on a personal project that needed a separate, offline linux environment. I tried various different schemes to run what I was doing in the confines of my laptop and I couldn’t make what I was working on actually achieve my goals. So… I bought a Raspberry Pi Zero W and a “Solderless Zero Dongle“, with the intention of running Docker containers on it… unfortunately, while Docker runs on a Pi Zero, it’s really hard to find base images for the ARMv6/armhf platform that the Pi Zero W… so I put it back in the drawer, and left it there.

Roll forwards a month or so, and I was doing some experiments with Nebula, and only had an old Chromebook to test it on… except, I couldn’t install the Nebula client for Linux on there, and the Android client wouldn’t give me some features I wanted… so I broke out that old Pi Zero W again…

Now, while the tests with Nebula I was working towards will be documented later, I found that a lot of the documentation about using a Raspberry Pi Zero as a USB gadget were rough and unexplained. So, this post breaks down much of the content of what I found, what I tried, and what did and didn’t work.

Late Edit 2021-06-04: I spotted some typos around providing specific DHCP options for interfaces, based on work I’m doing elsewhere with this script. I’ve updated these values accordingly. I’ve also created a specific branch for this revision.

Late Edit 2021-06-06: I’ve noticed this document doesn’t cover IPv6 at all right now. I started to perform some tweaks to cover IPv6, but as my ISP has decided not to bother with IPv6, and won’t support Hurricane Electric‘s Tunnelbroker system, I can’t test any of it, without building out an IPv6 test environment… maybe soon, eh?

Kinda-works – RNDIS with g_ether

Most of the content you find when you start looking into the USB-OTG mode content on Raspberry Pi Zeros is a switch to add to your /boot/cmdline.txt that adds, after “rootwait“, this: rootwait modules-load=dwc2,g_ether (or possibly rootwait modules-load=dwc2,g_ether g_ether.host_addr=00:00:5e:00:53:01 g_ether.dev_addr=00:00:5e:00:53:02 which sets the MAC addresses of each end of the connection). Some also say to put a line which says dtoverlay=dwc2 in /boot/config.txt. I’ve done both!

While this does work, on Windows devices, you need to find a RNDIS 5 driver. It’s commonly suggested that you should use the Acer Ethernet/RNDIS driver, listed in the “Microsoft Catalogue“, here.

Also, if you wanted to investigate using the USB interface to also surface a serial port (for “Console access”) or a storage volume (for drivers, perhaps, or just because you’ve got space to burn on that MicroSD card?) then you can’t load these at the same time as g_ether. So, let’s look at our next option.

Introducing libComposite

In the Linux Kernel, a module called “libcomposite” is available to create a USB-OTG composite device, which will serve several “functions” on the same interface. I was largely, initially, driven by a post on isticktoit.net where the initial basis of these steps were put together.

In essence, you load the kernel module, which adds a new directory to the “configfs” filesystem in /sys/kernel/config/usb_gadget. In here you can create a new device (I called mine libComposite but you can call it whatever you want, and then this creates a series of paths for you, including the files idVendor, idProduct, bcdDevice, bcdUSB and bDeviceClass. Essentially, these telegraph to the operating system the USB port is connected to what kind of device this is.

In Windows, the combination of the idVendor (called VID in Windows) and idProduct (PID) are combined to determine what driver needs to be loaded. The bcdDevice (REV) value is used for caching what driver should be loaded for that device (hint, if you have an issue with the USB device, you can either increment that bcdDevice by 1, OR go into your registry editor, and find the path Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\usbflags\ and look for a “key” or folder named as the idVendor and idProduct and bcdDevice as one string, and delete it.)

So that the OS knows what to call the appliance you’ve added to the USB port, there’s another directory here – strings which has a series of hexadecimal numbered directories. Based on Microsoft documentation, 0x409, is the strings in English. So, in the path strings/0x409/ will be three files, serialnumber, product and manufacturer which contain how the device is described on the OS. Rather than generating my own serial number or concocting my own product string, I’ve used the serial number and model details baked into the CPU, which you can find in /proc/cpuinfo.

Right, so now we need to start looking at “Functions”. A function, in libcomposite terms, is a feature that the device tells the host it can perform, like being a serial adaptor, or a storage device, or in our case, a network adaptor. Each group of functions is added to a configs directory, and has a functions directory (like rndis.usb0 or ecm.usb0) linked into that configs directory. The functions directory holds certain configuration values, and the configs directory identifies allows the OS to pick between multiple (but, usually just one) function, based on some values stored in there.

Each “function” also needs to be told whether the host computer needs to power it, and also how much power it needs. These are stored in the configs/c.<id>/bmAttributes file and the configs/c.<id>/MaxPower file, respectively.

Works on everything but Windows – ECM/CDC with libcomposite

When USB was first starting up, the USB Implementers created a networking layer called “ECM” (or “Ethernet Control Model) as part of the “CDC” (or “Communications Device Class”) group of interfaces. This interface works across Mac and Linux appliances, but sadly doesn’t work with Windows (I’ll get to that in a tic). You’ll sometimes see the name “ECM” or “ECM/CDC” used.

To create an ECM device, we create /sys/kernel/config/usb_gadget/libComposite/functions/ecm.usb0 and then link that whole directory into /sys/kernel/config/usb_gadget/libComposite/configs/c.1/.

By default, ECM will provide a dynamic MAC address for both the host (the machine it’s plugged into) and device (the device being plugged in) unless these are specified by putting values into functions/ecm.usb0/host_addr and functions/ecm.usb0/dev_addr respectively.

So, this is a relatively simple set of configuration… relatively. Now, let’s dive into making Windows work!

Works on Windows… and probably everything else, but … meh, they’ll just use ECM/CDC

So, I mentioned before that the USB Implementers created ECM… well, about that point Microsoft was still being a bit… well, they had “Not-Invented-Here-Syndrome”, so they stuck some duck tape (TM) on an old DOS protocol called NDIS, made it work over USB and called it RNDIS, and put it into Windows Mobile devices… YEY! But it’s not compatible with ECM/CDC… BOO! Oh, and it’s proprietary, so you need to hack about with it to make it work, because the specs aren’t always implemented right…. HIIIISSSSSS!

Anyway, the g_ether thing we used before is RNDIS 5, which required a whole load of hacky, vendor specific scripts to make it work, and not think it was a Serial Adaptor, but libComposite has an RNDIS 6 implementation, which has a driver that ships with Windows, by default, now, so that’s nice. But it needs a bit more config.

For starters, you need to put some magic strings into extra files to make this work – notably, you need to create a directory called os_desc/interface.rndis and stick two files called compatibility_id and sub_compatibility_id in there, which have the strings RNDIS and 5162001 in them, you also need to put the string RNDIS into configs/c.2/strings/0x409/configuration.

Wrapping up libComposite

So, once you’ve got your configuration values loaded up, you now need to tell your the Raspbian OS to use these values with the host OS. How do you do this? You read the values in a magic directory – /sys/class/udc into a file under the libComposite gadget config path – UDC. And then the OS offers this to the host OS.

If you need to back out what you’ve done for some reason, write an empty string over that UDC file, and libComposite will remove it from the host system.

Loading libComposite each boot

Oh yes, did I not mention? There’s nothing which stores these values between boots, so instead you need to run a big script that each boot, so that it’ll offer itself up to the host OS. How do we do that? Why, yes, a SystemD Service Unit.

SystemD Service Units are basically how to you tell Linux what scripts to run each time it starts up. It also handles capturing any text output and can start things after other things have booted, and so on. It’s quite nifty really.

This unit file is pretty straightforward, so I’ll just paste it in here:

[Unit]
Description=Start libComposite

[Service]
Type=oneshot
ExecStart=/opt/lib_composite.sh

[Install]
WantedBy=multi-user.target

Basically, this says “run the script in /opt/lib_composite.sh every time this service is invoked”, and it also says “make sure this is run before you tell the OS it’s booted and ready for users to log in”.

But, where do you put this? I’ve elected to put it in /lib/systemd/system/libComposite.service and then I link that to /etc/systemd/system/multi-user.target.wants (which is what running systemctl enable libComposite.service does)

And what happens now? Well, if you were to boot your system just with this, you’d end up with two interfaces with no configuration on them – /dev/usb0 and /dev/usb1. That’s no good… we need to make these have IP addresses!

Setting up the interfaces with /etc/network/interfaces

So, because we’ve defined the ECM/CDC interface as c.1 and the RNDIS interface as c.2, the ECM interface will be usb0 and the RNDIS usb1. You could use NetworkManager or NetPlan, but I’m not sure either of these are supported with Raspbian, so instead, I’m using the good old fashioned /etc/network/interfaces files.

We can put a per-interface file into /etc/network/interfaces.d/<interface name>, so the first one we’ll create is /etc/network/interfaces.d/usb0:

allow-hotplug usb0
iface usb0 inet static
  address 192.0.2.1
  netmask 255.255.255.240 # AKA /28 or 14 usable addresses

I’ll create the same file for usb1 but set the address to 192.0.2.17 instead.

Now, if we were to boot this, the Pi end would get an IP address, but the Host wouldn’t. To fix THAT, we need a DHCP server, and DNSMasq is perfect for that.

DNSMasq

DNSMasq is a pretty deceptively simple application for offering a DHCP and DNS service. We need to configure this in two places, /etc/dnsmasq.conf and /etc/default/dnsmasq. Let’s get the main config sorted out first – /etc/dnsmasq.conf

In here, we tell it what interfaces to use, like this:

interface=usb0
interface=usb1
bind-interfaces

We also need to tell it what range of IP addresses to serve:

dhcp-range=usb0,192.0.2.2,192.0.2.14,2h
dhcp-range=usb1,192.0.2.18,192.0.2.30,2h

And lastly, what default route to serve over this… hmm, well, here we get to a bit of a crossroads. You see, this device needs, currently, to just terminate connections – while the USB transfer speeds are probably fast enough on my network for general purpose browsing, my WiFi or Wired network connections are faster, so for now, this is going to advertise that there’s no default route for this connection, like this:

dhcp-option=option:router

If we wanted to advertise default routes to this device, we could advise each route like this:

dhcp-option=usb0,option:router,192.0.2.1
dhcp-option=usb1,option:router,192.0.2.18
dhcp-option=usb0,option:router,192.0.2.1

And the last thing we need to do, is to tell DNSMasq not to bind to the “lo” (AKA “loopback”) interface, because we’re still using systemd-resolve service on here, which runs a DNS service on 127.0.0.53. So we need to edit /etc/default/dnsmasq and add this line in:

DNSMASQ_EXCEPT=lo

That means “bind to all the interfaces, except for the interface called lo“.

And now we can start DNSMasq… Except… OH NO, we need to control when DNSMasq starts on boot, because it relies on those usb0 and usb1 interfaces being there. So now we need to make some changes!

You see, on Raspbian, when you install dnsmasq, it puts a systemd unit file in (which is great), but it also creates a “SysV” init script in, and the systemd unit file calls the SysV file… except when you do systemctl enable dnsmasq it says “Aha, I’ve found the SysV file /etc/init.d/dnsmasq, and I will use that in preference over the /lib/systemd/system/dnsmasq.service file”… So we need, first, to stop /etc/init.d/dnsmasq, rename it to something else, then tell /lib/systemd/system/dnsmasq.service to use the renamed init.d script, and then enable that. Whew. We do all that with a script called “package_postinstall.sh“ and call that from a SystemD Service Unit file… again.

Setting up a WiFi connection

In order to install DNSMasq, as well as do anything else we need to do on this device that needs network, we need to make a wpa_supplicant file in /boot. This needs to store the SSID and PSK (“Pre-Shared Key”) for the WiFi Access Point. This file looks like this:

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
country=GB

network={
  ssid="exampleSSID"
  psk="abc123def456"
}

On the first boot, this file will be moved into the right place on the OS (/etc/wpa_supplicant/wpa_supplicant.conf), and removed from the /boot directory.

SSH Service

During the initial builds, we need an SSH Server to make sure we have everything we need. To do this, we put a SSH file into /boot, like we do with the wpa_supplicant file. This file can be empty however.

Wrapping it all up

So, now I have a set of post-install scripts in a Github Repo, here: https://github.com/JonTheNiceGuy/rpirouter/tree/StateInterfaceUpOnly and I run these with ansible-playbook, straight after I’ve written my image to the SD card.

And because it’s all nicely modularized ready for Ansible… I might be able to make something else from this later 😉

Featured image is “raspberry pie” by “stu_spivack” on Flickr and is released under a CC-BY-SA license.