andhop.dev lives on a Raspberry Pi 5. Can it automatically scale to handle 1 million TPS? No, a friend tested it for me and the little Pi caps out at a few thousand TPS. Is it distributed across the globe so all customers have sub 10 ms access to it? No, it’s just a Raspberry Pi sitting under my TV.

In the age of cloud computing it’s refreshing to take a step back and look at what it takes to run everything yourself. So let’s get started building like it’s the 2000s:

  • Self hosted
  • No frameworks

Domain name registration

I started by buying andhop.dev from Cloudflare (AWS Route53 doesn’t support the dev Top Level Domain (TLD)). The process to signup and pay was painless and they seem to have a good API (with accompanying documentation).

DNS

Okay, I said run everything yourself, but DNS is one place where it’s really hard to DIY. I could run my own nameserver, but there’s two problems:

  1. it kind of makes DNS redundant, once you know where the nameserver is (at my home IP) you know where andhop.dev is.
  2. Cloudflare doesn’t support changing the nameserver for domain names registered with Cloudflare.

So I’ll sticking to Cloudflare for DNS.

Once you have a DNS record you need to point it somewhere. This is where the big cloud providers will try to lure you toward their fancy services with 9 9’s of availability and edge locations around the world. Here is where our resistance begins for real. We’re going to self host everything else and point andhop.dev at my home network.

I use a simple systemd service which runs every 5 minutes to check my current IP and if it’s changed update Cloudflare, the basic idea is:

RECORD_NAME="andhop.dev"
CF_ZONE_ID="something_here"
CF_API_TOKEN="something_secret"
RECORD_ID=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/${CF_ZONE_ID}/dns_records?type=A&name=${RECORD_NAME}" \
        -H "Authorization: Bearer ${CF_API_TOKEN}" \
        -H "Content-Type: application/json")
NEW_IP=$(curl -s --max-time 10 http://checkip.amazonaws.com/)
curl -s -X PUT "https://api.cloudflare.com/client/v4/zones/${CF_ZONE_ID}/dns_records/${RECORD_ID}" \
   -H "Authorization: Bearer ${CF_API_TOKEN}" \
   -H "Content-Type: application/json" \
   --data "{\"type\":\"A\",\"name\":\"${RECORD_NAME}\",\"content\":\"${NEW_IP}\",\"ttl\":300,\"proxied\":false}"

Home Network Setup

The internet is vast and scary and we’re about to invite strangers in. Getting traffic from the internet to a Pi safely takes a bit of plumbing. The first step involves getting all the traffic to my router, thankfully CenturyLink is cool and put a nice sticker on their router with it’s IP address (192.168.0.1), username, and password. Then it just took some poking around to find the right settings. In my case it was a setting called “transparent bridging”.

The CenturyLink router is now forwarding all packets to my UniFi router, and by default the UniFi firewall drops all incoming connections, it only allows data in for established connections that devices on my LAN initiated. Let’s change that:

  1. Create a “hazmat” VLAN for the web server Pi
  2. Update the switch to put assign the Pi’s port to the vlan
  3. Give the Pi a static IP
  4. Allow the rest of my network to connect to the hazmat vlan, but not the other way around
  5. Update the router to forward all port 443 traffic to the Pi’s IP. The UniFi controller automatically updated the firewall rules to allow new connections on port 443 into the hazmat vlan

Pi

The Pi itself is set up to be as boring and secure as possible:

  1. Simple Rust server listening for new connections and serve static content
  2. The rust server is started using a systemd service running as an untrusted user
  3. Run the service in a read-only chroot

Server software

Following the design principals of no (minimal) frameworks this site is a custom Rust server, but I’m still using hyper for HTTP and rustls for TLS. The main pages like andhop.dev are html files that get included into the rust binary at build time pub static INDEX_HTML: &str = include_str!("/processed_html/index.html"); this way the running process doesn’t need to access the filesystem to serve a request, everything is already loaded into memory. This also means all routes are statically defined at compile time to completely stop any directory traversal attacks.

The main site pages have minimal templating support to handle some simple data like 905547. These variables in the HTML get replaced whenever a request is served. In the future I’d like to pre-compute this so serving a request is even faster and just involves writing a pre-generated string to the socket.

Blog posts like this one are compiled from Markdown files at build time to static HTML pages. The HTML pages are also included into the Rust source code like the homepage.