One of my favorite things about the world is that people keep making amazing stuff.

Tailscale is amazing stuff, that lets you make a lightweight, NAT-traversing, Wireguard-based mesh VPN with almost zero work. Guix is more amazing stuff, that lets you immutably describe system configurations, deployments, and virtual machines. And, with some work, you can use both together to make more amazing stuff!

My new favorite way to run self-hosted services is to use Guix System virtual machines, connected by Tailscale. Guix lets you easily spin up hermetic hosting environments, and Tailscale lets you trivially and securely connect to those hosts. This post explains how to do it, using the example of running an IRC bouncer. I'll be assuming that you're at least a bit familiar with Guix and with Tailscale, and that you have a server running Guix System.

Using Tailscale in Guix

The first thing to get set up is tailscale-in-guix. While the tailscale client is free software, and therefore fair game to be included in guix, it hasn't been packaged (yet!). You'll want to set up a personal channel. My personal channel is on Github, but I highly disrecommend using it directly, because I'm constantly breaking it. First, you'll want to add a package for tailscale, possibly like this one [^1]. And second, you'll also need to define a service-type (maybe like this one) so that you can easily add tailscale to your system descriptions.

Once you've added these things to your channel and pulled it, you should be able to get tailscale running on your host by adding your tailscale packages to the system packages list, and instantiating a `tailscaled-service-type` service (or whatever you called it) in your system's `services` list. Try running `tailscale up --ssh` to check that everything works!

Setting up a virtual machine to run your service

Now we want to set up a virtual machine to host our service. I prefer to use virtual machines over (e.g.) containers, because it's easier to understand their security properties, and to limit their resource usage. You probably want to start with the guix "bare-bones.tmpl" system template, as that contains the basic things you'll need to run a virtual machine. On top of that config, you'll want to add your service declaration. In this case, there's no `pounce` service included with guix, so I wrote a very simple one and added it to my channel. Then, I included that from my system description file.[^2] You should also include the tailscale service!

Before you build your VM, you'll also want to figure out where on the host you want to store any stateful components of your application. My server happens to have a massive raidz2 ZFS pool at `/tank`, so I'm storing all my state there, but you could use basically any location you have write access to.

What do you need to store this way? Anything that you don't want to get wiped out when you update the virtual machine. In my case, I'm treating the `pounce` configuration as mutable state, as well as its log and state files. And don't forget about tailscale's state! Given my service description, tailscale's state lives in `/var/lib/tailscale`. Finally, you probably want to store `ssh` host credentials, which by default live in /etc/ssh on a Guix System (otherwise, they'll get rebuilt whenever you rebuild the vm, causing your ssh clients to complain).

Got all that ready? Awesome, now let's write a script that sets up or updates your virtual machine. The script I use is available here, but the gist is that you want `guix system vm` to create a virtual machine for you, and to mount the required state directories from locations on the host. Then, you may want to run the VM, to let yourself look around a bit and see if everything is as intended, and to set up initial state: You probably want to run `ssh-keygen -A` and `tailscale up --ssh`, and in my case I also needed to run some `pounce` setup commands and `tailscale cert`, in order to generate SSL certificates for pounce to use. Finally, you should add a symlink to the result of `guix system vm` to your system's garbage collection roots, so that it doesn't get cleaned up the next time someone runs `guix gc`. I've done this by symlinking to that file from /var/guix/gcroots/pouncevm.

You'll probably need to fiddle with this a bit to get it to work for you, but once the state directories are set up and mounted properly, updating the vm shouldn't break anything, and you should have a self-hosted [something] that you can access over your tailnet! I highly recommend running your vms as a nonprivileged user, by the way, as doing otherwise could let a buggy or compromised service destroy your system (mainly because /gnu/store is mounted inside the virtual machine, and so if you run the VM as root it might write over your system store files!). In order to do this, you probably need your user to be in the `kvm` group.

Because the virtual machine shares its /gnu/store with the host, your new VM takes up far less space than you might expect! On my system, the disk image is just 2MiB! All the heavyweight stuff (qemu, the kernel, initrd) is shared among all my up-to-date VMs, and the state files all live elsewhere and can be managed directly from the host! You can also choose how much memory and how many cores your service is allowed to use. In my case, I currently stick with the defaults (512MiB of memory, 2 threads).

Running your VM as a user service

At this point, you can consider yourself done if you want. You can now run your VM, connect to it with tailscale and ssh, and everything is peachy. But what if you want the VM to be launched in the background on boot? This can be tricky - the GNU Shepherd (the init system used by Guix System) doesn't provide a trivial way to do this for a user service.

The solution I've come up with is to have a system-level shepherd service that runs at boot, which in turn loads a shepherd for my user. You can see the shell script I use for that, and the service definition. From there, it's relatively easy to follow the Shepherd manual to create a user service that runs the script that you symlinked to a gc-root!

Figuring this all out took me a substantial amount of effort; I hope some of it can be useful to you! Guix is, imo, an amazing project, but its community is tragically small, so finding information not contained in the manual can be a challenge! Several people helped me along the way, including nckx on libera.chat, and Skyler Ferris on the help-guix listserv.

[1] Note: I am not an expert at building stuff for guix! These configs were pieced together based on random sources from around the internet whenever i couldn't figure it out based on the guix docs, and it's likely that they contain bugs!

[2] The configuration here is mostly not done in the system description - I got lazy! If you want to make everything truly immutable, you'll want to add configuration options to the service description, and then instantiate those options in the system description. Instead, here I simply add the configuration directory to the stateful parts of the system (i.e. the parts that get mounted from the host).

I've been exploring Thunderbird[^1] today, and learned it had a major redesign some time in the last few years[^2]. Not only that, but it now seems pretty acceptable as a feed reader - I'm not sure why, but in the past it would often fail to import feeds. And it even has IRC support! I'm considering using it as my one-stop "technical part of the internet" reader. And since I've started using Posthaven as a blogging platform, it even works as a blogging tool! (or, at least, if this post goes up nicely, that's a good sign)

[1] After about 15 years of using GMail, followed by about six months of Mail.app. Recently, I wanted to sign up to some open-source projects' mailing lists, when I noticed that all prior attempts to join such lists had ended in hasty unsubscriptions. Reflecting on this, I noticed that I really dislike the way GMail deals with threading on large lists, clumping the whole discussion into one big stream. And so I went on a quest for a decent compromise between "usable without too much faff" and "technically decent". I briefly considered Mutt, but at this point in my life I don't know if I want to spend the effort on learning a whole new TUI when I can easily avoid it. Neovim and tmux are sticking with me, but best of luck convincing me it's worth switching to a TUI-based music player or something.

[2] My initial reaction was "this can't be good", but after a few hours I actually quite like it.

Got back from my first Burning Man earlier this week. Some thoughts about it:

Art

People say that Burning Man has lots of "art". The word "art" doesn't cut it as a description. Black Rock City felt, to me, like Earth's only spiritport city. The city proper is mundane and organized, despite many anomalies. You'd never mistake it for a "normal" city, but it's not too far from the midpoint between an uncovered bazaar in the desert, and Deep Space Nine.

Photo credit: Chayna Girling on Flickr

The playa outside of the street "grid", though, is like a spirit vessel harbor. There are massive moving mutant vehicles, huge otherworldly "sculptures" that feel like living beings, ghostly wandering tea shops, drone swarms bearing messages, intricate meaning-laden towers, swarms of humans dancing mindlessly to leviathan heartbeats. It's shockingly dense with mindfruit. Last month it was emptiest desert. Next month it will again be emptiest desert.

I didn't take any drugs while there; nonetheless I found the experience totally overwhelming.

Photo credit: Duncan Rawlinson on Flickr

Culture

Burning Man's culture is diverse. I've heard there are 70,000 people present.

I observed some burner archetypes:

• "Old hand": 50-70, Someone who's been coming since like 1995. Friendly, chill.
• "Euro raver": 30-50, usually from Europe, goes to lots of rave-y events and loves the playa art. Friendly.
• "Influencer": 20-35, takes a bunch of selfies, always looks shiny. Seem to disappear if the weather is bad. Less friendly.
• "Techie": 25-45, probably lives a pretty active lifestyle within the confines of a career as a programmer. Friendly but also a bit aloof.
• "Kinkster": 35-50, here for sex stuff, of which there's plenty. Friendly (not in a gross way). Friendly (also in a gross way).
• "Radical leftist": 20-35, here for political reasons. Anticapitalist, social/environmental justice. Mostly wants to do things that are advancing their cause. Not actively friendly, but not actively unfriendly unless you do something objectionable.

Not everyone fits one of these stereotypes, but many do, and lots fit several. I'm sure there are dozens more archetypes; this was my first burn, and the second half was dampened a bit (literally).

In some respects, I felt very at home there. In others, very alienated.

There are 10 "principles", allegedly set down as a description of the community rather than a normative requirement: Radical Inclusion, Gifting, Decommodification, Radical Self-reliance, Radical Self-expression, Communal Effort, Civic Responsibility, Leaving No Trace, Participation, and Immediacy.

Most of these principles resonated extremely strongly with me. Several of them are something like "the best parts of Scouting": Radical self-reliance, civic responsibility, leaving no trace. And others are something like "fuck judgmentalness", which is an ethic I love: Radical inclusion, radical self-expression.

Some of the principles strike me as "interesting", without feeling like I fully understand or endorse them. Gift-giving and Decommodification are examples. I like the idea of building "weird" economies, on top of human values other than resource acquisition; I first came across this idea in the book Why Not Socialism?, and the idea has stuck with me.

But there's something kind of ironic about the tens of millions of dollars that got spent in the "default world" economy, in order to allow this brief glimpse at a thing that probably wouldn't work if scaled up, at least not naively. From outside, Burning Man looks a lot like one of the biggest consumption-festivals in the world. Nobody farms in black rock city. I don't think I saw anyone making things that people were likely to use in their "default world" lives. Certainly people take experiences and ideas back with them, and I'm sure that the value of that is huge. But some part of me wants to be like "call me when you can reach a critical production temperature, and can bring material gifts back for everyone else." I wonder what a cross between Burning Man and the Oneida Community would look like.

And some of the values feel like they kind of "don't parse", for me. I'm not sure what they actually mean, or what they're there for: Immediacy and Participation, in particular, don't grab me, despite seeming to be some of the most central principles for others.

Preparations

I loved the challenge of getting ready on short notice. I camped solo with a friend, not with an established camp. This was awesome because it let me try lots of fun stuff.

We opted to sleep on an air mattress in the back of my car, originally because this would let us better control the air quality in the sleeping area (as my car is relatively airtight, compared to a tent). I modified a car cover to put some MERV-16 filter fabric over the windows, so that they could be rolled down to give us some ventilation without letting in too much dust. Then I bought a ridiculous number of magnets, which I used to securely fasten the car cover to the car's body. This worked great; I was initially worried that the magnets might not be strong enough, but in the end I used something like 100 strong magnets and I think none of them came loose. Sleeping in the car was also the right call given that it ended up raining. I would not have wanted to be stuck camping in a tent on top of a foot of wet modeling clay.

We used aluminum netting for our shade structure, staked to the ground with U-shaped rebar. This also worked great, though if I do it again I'll want to ensure that the shade is high up enough to stand up properly underneath.

I borrowed some nice solar panels from my housemate, and bought a big AGM battery and inverter to connect them to. When it got wet out, I was honestly a little afraid of this equipment, but we disconnected everything and put it into a watertight plastic bin before the rain started.

I had hoped to set up a radio link to the Burning Man internet backbone, and I came quite close to succeeding at this goal. A couple times, I successfully got my radio talking to the backbone's radio with a low bitrate (you can see my improvised radio "tower" in the image above). But despite that, I was unable to get any actual packets onto the internet, and this ended up being a waste of time. I think it would have worked if we were closer to center camp. Anyway, it was fun to try to get it working.

Next Time

I'd love to come back again. I'd love to spend more than a couple of weeks preparing (I decided to go this year at the last minute), and maybe even build a contribution to the crazy spirit world. Hopefully next year.

Some are concerned about the air quality at Burning Man. There's lots of dust, and the dust is made of hazardous materials: gypsum, silica, and iron. This will be my first year, and I wanted to learn about the risks. These are some things I found. TL;DR: I won't be skipping the burn due to the air quality issues, but I will be trying to keep my sleeping quarters relatively free of airborne dust, and aiming to wear a P100 mask when the wind gets above about 20mph.

I considered two main types of risk:

1. Silicosis
2. General risks from dust inhalation

Silicosis

Silicosis is a very unpleasant and potentially deadly lung problem that can occur due to exposure to large amounts of silica dust. It kills tens of thousands of people per year worldwide, mostly due to occupational exposure. However, it only kills about a hundred people per year in the United States. This difference is partly because here, workers are usually not exposed to silica levels above thresholds set by OSHA.

So how bad is the exposure to silica dust on the playa, compared to occupational exposure limits? In a 2019 Bureau of Land Management report on "Public Health and Safety at the Burning Man Event", the authors note:

OSHA thresholds vary from the EPA and are based on exposure during an 8-hour shift over the course of a 40-year career...

An air quality study was performed by government industrial hygienists during the 2018 Burning Man Event. The study found six samples exceeding the OSHA permissible exposure limits (PELS) for respirable crystalline silica and three exceeded OSHA PELS for total respirable dust. Crystalline silica is a contributing factor to silicosis of the lung and a known carcinogen naturally occurring on the playa surface... All samples exceeding PELS occurred with winds in excess of 18–20 miles per hour. It is recommended that all exposed employees use an N95 Respirator when winds are in excess of 18–20 miles per hour and reduce the use of open-air vehicles.

Based on this, I think it seems unlikely that attending Burning Man for a week every year poses a risk comparable to e.g. working in an industry with large amounts of exposure to silica dust, especially if one takes precautions during the dustiest time.

Dust Inhalation

Separate from silicosis, high levels of particulate matter of any kind pose a health hazard. You can find data from 2017 showing hazardous dust levels (both pm10 and pm2.5) basically the whole week. Allegedly there was a nearby wildfire that may have made these numbers look worse than normal, but let's assume they're representative. The concentrations of both pm2.5 and pm10, throughout the week, put the AQI solidly in the hazardous range at many times, once even exceeding the pm10 AQI scale. This is concerning! However, a quick check indicates that the health hazard (at least due to pm2.5) is about as bad as smoking 1 cigarette per hour. This is bad (equivalent to about 1 micromort per hour), but it's manageable. You can improve matters by wearing a protective face covering - especially a p100 or n95 mask.

Taken together, these hazards suffice for me to take extra measures to protect my lungs while at Burning Man, but not for me to skip the event entirely. My plan is to wear a p100 mask whenever convenient (especially when the wind is up), and to sleep in my car with MERV16 filter fabric taped over the windows.