simister.xyz ← Back

Field notes

A command centre made of ordinary parts.

Three machines that are rarely all awake at once, a job queue that is just a directory, and a set of decisions that were mostly about what happens when something is switched off.

Right now

hub pc phone

reading…

The system

The job queue is a directory

Work is a JSON file. A machine takes a job by renaming it from one folder into another. There is no broker, no database and no lock server, because a rename on a single filesystem is already atomic — two machines cannot both win it.

pending/  ──rename──▶  claimed/  ──rename──▶  done/
                          └─ unfinished after 30 min ─▶  pending/

The reaper that returns abandoned jobs measures age by ctime, not mtime. A rename updates ctime and preserves mtime, so ctime is when the job was claimed and mtime is when it was created. Reading the wrong one means a stuck job is never returned.

The app store outlives the machine that builds it

The desktop signs Android builds and generates the repository index. This server only serves the resulting static files — it has never run the build tooling and never needs to. The store stays up while the machine that fills it is switched off, which is most of the time.

Neither machine can sign a release alone

The signing key lives on the desktop. Its passphrase lives here, readable only by root. A build pulls the passphrase across the private network at the moment it is needed.

It is handed to the signer on stdin, never as an argument — arguments are visible to every other process on the machine.

Certificates are issued on demand, behind a gate

Every name under this domain points at this box, so any stranger could point a hostname here and trigger a certificate request. A small service checks each request against an allowlist before issuance is allowed.

It fails closed: an unreadable or empty allowlist approves nothing. Without it, a few thousand requests would exhaust the weekly issuance quota and take every site here offline at once.

One sign-in, two kinds of credential

A single service answers “may this request through?” for every protected site here. A browser gets a real login page and a signed session cookie scoped to the whole domain, so signing in once covers all of them.

The same accounts also answer HTTP Basic, because the F-Droid Android client speaks Basic and nothing else. Dropping it for a nicer login page would have silently broken every phone already subscribed to the repository — the failure would have shown up as apps quietly not updating, which is the kind that takes weeks to notice.

Almost nothing is reachable from the internet

Two ports are open to the world and both belong to the web server. The desktop’s remote access listens only on its private address, and the machines address each other over a private network that is not port-forwarded anywhere.

Three times it looked right

An access rule that returned 200

Looked like
The config validated cleanly. caddy validateValid configuration.
Actually
The web server sorts directives by type, and handle sorts before respond. The 403 was written last and ran never. A blocked address got a 200.

A node list that was always empty

Looked like
The dashboard loaded. No error, no failed request, nothing in the log. Just no machines listed.
Actually
A hardening flag, RestrictAddressFamilies=AF_INET, allowed network sockets only. The status command talks to its daemon over a unix socket.

A worker that shredded every job it touched

Looked like
Jobs moved pending → claimed → done. Every transition in the state machine was correct.
Actually
Every file arrived 0 bytes. A redirect with nothing on stdin had emptied them. No state the system tracked could show it — only file size could.

None of these crashed. That is the part worth keeping: the failures that cost the most time all looked exactly like success.

Underneath

Server
Ubuntu 24.04 on 2 ARM cores and 11 GB, Oracle Cloud UK South
Web
Caddy, automatic TLS, one site file per subdomain
Network
Tailscale — every machine addressed privately, nothing port-forwarded
Services
systemd units, hardened and unprivileged
Code
Python standard library and bash. No framework, no build step
Notes
51 markdown files, 230 links between them, snapshotted into git