Cloud re-init

Thanks to my recent experience with not having a proper work machine I decided I wanted to set up a little RaspberryPI that I could use for coding in Go, Rust, Python etc. even from my iPad.

Ideally I’d connect to it via ethernet but as a fallback it should simply open a tunnel to one of my publicly reachable servers. SSH for the win.

At a recent trip I spent a bit of time to analyse the captive portal of the environment there and created a little script that would simply log into the session every couple of minutes.

I was also reminded that the whole idea of cloud-init scripts on things like the RaspberryPIs was that they’d only be applied once. I somehow hoped that at least the network-config would be applied with every boot by the configuration shipped with Ubuntu Server. Took me a day to figure that out since I didn’t have an USB keyboard with me 😅 Since I had missed the opportunity to configure eth0 properly I needed to reset the whole device, but that took only 30 minutes or so thanks to Ansible playbooks I already had in place.

How does it work?

Anyway, how is cloud-init configured to take the initial config from the system-boot partition of the SDCard and how can configure it to read that configuration for every boot? Since I’m especially interested in the network configuration, I’ll focus on that part here that also involves the netplan configuration.

The initial configuration is taken from the system-boot partition which is the first one on the SD card:

(parted) print all
Model: SD SC128 (sd/mmc)
Disk /dev/mmcblk0: 128GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number  Start   End    Size   Type     File system  Flags
 1      1049kB  269MB  268MB  primary  fat32        boot, lba
 2      269MB   128GB  128GB  primary  ext4

Sadly, it seems like parted cannot display the label of a partition, so let’s do that with e2label:

$ e2label /dev/mmcblk0p1
e2label: Bad magic number in super-block while trying to open /dev/mmcblk0p1
/dev/mmcblk0p1 contains a vfat file system labelled 'system-boot'

When Ubuntu boots, it also starts a cloud-init service which takes its configuration from the files inside the /etc/cloud folder. Among these there is also /etc/cloud/cloud.cfg.d/99-fake_cloud.cfg which has the following content:

# configure cloud-init for NoCloud
datasource_list: [ NoCloud, None ]
datasource:
  NoCloud:
    fs_label: system-boot

This defines, that cloud-init should use only the datasource NoCloud which is the filesystem on the system-boot partition. Mounting this partition (it is automatically mounted into /boot/firmware) reveals the same content as you’d get if you plug the SDCard into a Mac or Windows PC:

$ cd /boot/firmware
$ ls
...
config.txt
...
network-config
...
usercfg.txt

For me of particular interest is the network-config file. When does cloud-init load that and could I convince it to always load it during booting and not just for the first time?

Let’s say, we add another WiFi configuration to the network-config for the access point “HotSpot” without any password. The goal is, that this new configuration shows up inside /etc/netplan/50-cloud-init.yaml which is generated by cloud-init with the information from the network-config file of the boot partition.

It’s everything or nothing!

Sadly, I couldn’t find a proper way to only re-run only the network-part of cloud-init on every boot. cloud-init clean removes all locks and semaphores but that’s also the problem: it removes all state which also means that the user-scripts etc. are executed again.

When I run cloud-init clean and reboot the machine the network configuration is updated. If the user-data part of the cloud-init configuration is empty, then that’s also it. I can live with that.

What about host keys?

The removal of all state in cloud-init also means that the host keys will be regenerated during the next boot. This was extremely annoying and so I removed the ssh module from the modules-configuration inside /etc/cloud/cloud.cfg:

# The modules that run in the 'init' stage
cloud_init_modules:
 - migrator
 - seed_random
 - bootcmd
 - write-files
 - growpart
 - resizefs
 - disk_setup
 - mounts
 - set_hostname
 - update_hostname
 - update_etc_hosts
 - ca-certs
 - rsyslog
 - users-groups
 #- ssh

Auto-cleaning

So now I’m left with one point on the todo list: How can I run cloud-init clean after every successful run of cloud-init?

For me the easiest approach was to just create a oneshot service that should be run after cloud-final:

[Unit]
Description=Reset cloud-init state
After=cloud-final.service cloud-init.service cloud-init-local.service

[Service]
Type=oneshot
ExecStart=/usr/bin/cloud-init clean
KillMode=process

StandardOutput=journal+console

[Install]
WantedBy=cloud-init.target

That’s pretty much it! Now I have a RaspberryPI with an SDCard where I can configure the network before every boot. This means that I can easily hang it into pretty much any WiFi without first having to boot and configure it via ethernet 🙂

It’s probably not the cleanest approach to get there but I was too lazy to dig deeper or write a custom system that only exposed the network configuration. The SDCard is, in any case, the biggest attack surface here no matter what part of the cloud-init configuration is read from it (hint: everything in any case). As for private data: LUKS FTW 😅

As a next step, I want to finalise the SSH tunnel setup so that I have a known entry point. More on that hopefully another time 😅