One of my favorite things about the world is that people keep making amazing stuff.

Tailscale is amazing stuff, that lets you make a lightweight, NAT-traversing, Wireguard-based mesh VPN with almost zero work. Guix is more amazing stuff, that lets you immutably describe system configurations, deployments, and virtual machines. And, with some work, you can use both together to make more amazing stuff!

My new favorite way to run self-hosted services is to use Guix System virtual machines, connected by Tailscale. Guix lets you easily spin up hermetic hosting environments, and Tailscale lets you trivially and securely connect to those hosts. This post explains how to do it, using the example of running an IRC bouncer. I'll be assuming that you're at least a bit familiar with Guix and with Tailscale, and that you have a server running Guix System.

Using Tailscale in Guix

The first thing to get set up is tailscale-in-guix. While the tailscale client is free software, and therefore fair game to be included in guix, it hasn't been packaged (yet!). You'll want to set up a personal channel. My personal channel is on Github, but I highly disrecommend using it directly, because I'm constantly breaking it. First, you'll want to add a package for tailscale, possibly like this one [^1]. And second, you'll also need to define a service-type (maybe like this one) so that you can easily add tailscale to your system descriptions.

Once you've added these things to your channel and pulled it, you should be able to get tailscale running on your host by adding your tailscale packages to the system packages list, and instantiating a `tailscaled-service-type` service (or whatever you called it) in your system's `services` list. Try running `tailscale up --ssh` to check that everything works!

Setting up a virtual machine to run your service

Now we want to set up a virtual machine to host our service. I prefer to use virtual machines over (e.g.) containers, because it's easier to understand their security properties, and to limit their resource usage. You probably want to start with the guix "bare-bones.tmpl" system template, as that contains the basic things you'll need to run a virtual machine. On top of that config, you'll want to add your service declaration. In this case, there's no `pounce` service included with guix, so I wrote a very simple one and added it to my channel. Then, I included that from my system description file.[^2] You should also include the tailscale service!

Before you build your VM, you'll also want to figure out where on the host you want to store any stateful components of your application. My server happens to have a massive raidz2 ZFS pool at `/tank`, so I'm storing all my state there, but you could use basically any location you have write access to.

What do you need to store this way? Anything that you don't want to get wiped out when you update the virtual machine. In my case, I'm treating the `pounce` configuration as mutable state, as well as its log and state files. And don't forget about tailscale's state! Given my service description, tailscale's state lives in `/var/lib/tailscale`. Finally, you probably want to store `ssh` host credentials, which by default live in /etc/ssh on a Guix System (otherwise, they'll get rebuilt whenever you rebuild the vm, causing your ssh clients to complain).

Got all that ready? Awesome, now let's write a script that sets up or updates your virtual machine. The script I use is available here, but the gist is that you want `guix system vm` to create a virtual machine for you, and to mount the required state directories from locations on the host. Then, you may want to run the VM, to let yourself look around a bit and see if everything is as intended, and to set up initial state: You probably want to run `ssh-keygen -A` and `tailscale up --ssh`, and in my case I also needed to run some `pounce` setup commands and `tailscale cert`, in order to generate SSL certificates for pounce to use. Finally, you should add a symlink to the result of `guix system vm` to your system's garbage collection roots, so that it doesn't get cleaned up the next time someone runs `guix gc`. I've done this by symlinking to that file from /var/guix/gcroots/pouncevm.

You'll probably need to fiddle with this a bit to get it to work for you, but once the state directories are set up and mounted properly, updating the vm shouldn't break anything, and you should have a self-hosted [something] that you can access over your tailnet! I highly recommend running your vms as a nonprivileged user, by the way, as doing otherwise could let a buggy or compromised service destroy your system (mainly because /gnu/store is mounted inside the virtual machine, and so if you run the VM as root it might write over your system store files!). In order to do this, you probably need your user to be in the `kvm` group.

Because the virtual machine shares its /gnu/store with the host, your new VM takes up far less space than you might expect! On my system, the disk image is just 2MiB! All the heavyweight stuff (qemu, the kernel, initrd) is shared among all my up-to-date VMs, and the state files all live elsewhere and can be managed directly from the host! You can also choose how much memory and how many cores your service is allowed to use. In my case, I currently stick with the defaults (512MiB of memory, 2 threads).

Running your VM as a user service

At this point, you can consider yourself done if you want. You can now run your VM, connect to it with tailscale and ssh, and everything is peachy. But what if you want the VM to be launched in the background on boot? This can be tricky - the GNU Shepherd (the init system used by Guix System) doesn't provide a trivial way to do this for a user service.

The solution I've come up with is to have a system-level shepherd service that runs at boot, which in turn loads a shepherd for my user. You can see the shell script I use for that, and the service definition. From there, it's relatively easy to follow the Shepherd manual to create a user service that runs the script that you symlinked to a gc-root!

Figuring this all out took me a substantial amount of effort; I hope some of it can be useful to you! Guix is, imo, an amazing project, but its community is tragically small, so finding information not contained in the manual can be a challenge! Several people helped me along the way, including nckx on libera.chat, and Skyler Ferris on the help-guix listserv.

[1] Note: I am not an expert at building stuff for guix! These configs were pieced together based on random sources from around the internet whenever i couldn't figure it out based on the guix docs, and it's likely that they contain bugs!

[2] The configuration here is mostly not done in the system description - I got lazy! If you want to make everything truly immutable, you'll want to add configuration options to the service description, and then instantiate those options in the system description. Instead, here I simply add the configuration directory to the stateful parts of the system (i.e. the parts that get mounted from the host).