Article summary
I have long been a fan of OpenBSD, over two and a half decades after I first started using it. It’s a complete operating system maintained with a strong and smart design ethos. It makes sense to me in a way that makes every other operating system feel frankly kind of cobbled together.
Many folks know little of it. But the OpenSSH client they use every day came directly from the careful work of the OpenBSD developers. It’s just one small part of the system.
I maintain an OpenBSD server of my own, hosted by the wonderful OpenBSD Amsterdam. It was there I sought to host a little private Slack bot of my own.
httpd(8)
My personal website is served from that same OpenBSD server, in the old self-hosting tradition. I use the built-in httpd(8) server for everything—all four domains.
It’s perhaps a little strange to see a program called “httpd” referred to as “httpd(8)”, but that’s part of the OpenBSD culture as well. Everything in OpenBSD is documented by a manual page; httpd’s is in section 8.
I have server sections in my httpd.conf(5) covering each of my four domains, with several rules matching old URLs and redirecting to appropriate places in my site’s current layout. There’s also configuration to support acme-client(1), which maintains TLS certificates from Let’s Encrypt.
On this server, I started developing the Slack bot with uv and Slack’s Bolt library for Python. I got everything pretty well up-and-running with socket mode, which only makes outgoing connections, until I wanted to “install” my bot’s app on another Slack server.
Now I needed OAuth. Bolt supports OAuth, but it required me to move away from socket mode and run a server. My tiny little hobby project would create a server on port 3000, but it wasn’t HTTPS. I wanted to stick something in front of it besides.
I’ve been to this rodeo before, just not on OpenBSD. I went to the httpd.conf manual page and looked for the reverse proxy support I was used to from many other kitchen-sink web servers.
Hmm. There wasn’t any.
relayd(8)
httpd, as it says on its manual page, is actually based on relayd(8). relayd is not a web server, but a daemon that can redirect incoming network connections in many ways—not just at the network layer, but at the application layer, i.e. HTTP.
I knew of relayd, but hadn’t used it before. But I was finding in my searches that everyone used httpd in conjunction with relayd to solve similar problems.
It may seem kind of strange to split this out. httpd serves HTTP—why doesn’t it also perform the reverse proxy function? It could, but… relayd already does this, and a lot more, in the land of incoming connection routing.
That is, I think, part of the design that speaks to me so much. Programs on OpenBSD do one thing and do it very well. Doing one thing well also gives them space to be very flexible at it.
relayd can stick TLS in front of any protocol, as well as peek into HTTP requests and route them wherever they need to go. Which, it turns out, is exactly what I needed.
relayd.conf(5)
relayd.conf(5) is where we define relayd’s behavior. Lists of hosts we’ll be relaying to go in table sections. (I only need 127.0.0.1 here, but we could use more!)
table <www> { 127.0.0.1 }
table <bot> { 127.0.0.1 }
Next comes the protocol section, where I define the rules for processing HTTP. Like httpd.conf, it uses filter rules in the style of pf.conf(5) (for OpenBSD’s famous packet filter pf(4).) The section that routes traffic looks like this:
block
pass request header "Host" value "example.com" forward to <www>
pass request header "Host" value "example.net" forward to <www>
pass request header "Host" value "bot.example.com" forward to <www>
pass request header "Host" value "bot.example.com" path "/slack/*" forward to <bot>
These rules process top-to-bottom, and the last rule that matched wins. It’s a style that takes a bit of getting used to, but can be quite powerful once you get used to it.
Most traffic gets passed to a host defined in the “www” table. The Host header is also passed, so httpd will be able to figure out which domain the traffic is destined for.
The final rule is where the Slack bot magic happens, though. If the path, using globbing rules, matches something under “/slack”, it’ll get routed to the “bot” table instead.
A relay section wraps it all up and defines what port we should talk to the HTTP server on—port 81 (I’ll explain that in a moment!) for our normal webserver, and port 3000 for our Slack bot. The whole file looks like this, including loading the TLS key pairs for HTTPS and support for forwarding headers:
table <www> { 127.0.0.1 }
table <bot> { 127.0.0.1 }
http protocol "https" {
match request header append "X-Forwarded-For" value "$REMOTE_ADDR"
match request header append "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
tls keypair example.com
tls keypair example.net
tls keypair bot.example.com
block
pass request header "Host" value "example.com" forward to <www>
pass request header "Host" value "example.net" forward to <www>
pass request header "Host" value "bot.example.com" forward to <www>
pass request header "Host" value "bot.example.com" path "/slack/*" forward to <bot>
}
relay "https" {
listen on egress port 443 tls
protocol "https"
forward to <www> port 81
forward to <bot> port 3000
}httpd.conf(5)
I already had a configuration file for httpd, but I needed to modify it somewhat. It was no longer going to be answering HTTPS on its own, since that would now be relayd’s job.
There were a couple of things I needed to address. I normally issue a 302 Found when an unencrypted HTTP request comes in, to push it to HTTPS. But if I did that now on port 80, and had relayd forwarding traffic to port 80, I’d get stuck in a redirect loop.
So what I did instead was use port 80 for that redirect, plus serving up the HTTP-01 challenge for Let’s Encrypt:
server "*" {
listen on egress port 80
location "/.well-known/acme-challenge/*" {
root "/acme"
request strip 2
}
location * {
block return 302 "https://$HTTP_HOST$REQUEST_URI"
}
}
Then, each one of my virtual servers were modified slightly, to listen only on 127.0.0.1, port 81—and log forwarded addresses:
server "example.com" {
listen on 127.0.0.1 port 81
log style forwarded
# (rest of example.com configuration)
}
acme-client.conf(5)
Finally, I had a small change to make to acme-client’s configuration, so that Let’s Encrypt would work properly: renaming the certificates.
In the example acme-client.conf found in /etc/examples, the certificates are named “example.com.fullchain.pem”. relayd is loading the certificates now, and it’s pretty picky about the naming, so a small change was in order:
domain example.com {
domain key "/etc/ssl/private/example.com.key"
domain full chain certificate "/etc/ssl/example.com.crt"
sign with letsencrypt
}
And that was it! With acme-client in /etc/daily.local, everything worked together, and maintained my TLS certificates without any fuss.
curl(1)
The venerable curl can be used for lots of things, including lying to a web server about what hostname you used to reach it.
For example, to test httpd on port 81, normally only reachable via the TLS forwarding from relayd, I can do this (from the server, of course):
curl \
--connect-to 'example.com:80:127.0.0.1:81' \
-I http://example.com
When I ask curl here to connect to the webserver, it actually sends the connection to the local address, port 81 instead. This lets me peek behind relayd and check all is well.
This also works great for checking that my underlying setup is working and using the correct hostname even when someone like Cloudflare is proxying from the public Internet. I can ask for my own domain, but direct traffic locally to check the local certificate and configuration.
exit(3)
And that’s it. It may look like a lot, but it covers absolutely everything—and with clean separation of duties between relayd and httpd.
A setup like this is very flexible. relayd is doing the job it does best—routing incoming traffic—and httpd doesn’t need to worry about connecting to my little Slack bot. relayd, httpd, and the bot could even all live on completely different servers.
OpenBSD has always been a model of consistent, solid design and engineering. People who resonate with it love it. Back in the day when I used to port third-party programs to it, I tried many free operating systems. OpenBSD was the one that won my heart.