How a small network of sites — two public apps and the backend services they share — went from “someone SSHes in and runs a script” to “react with a checkmark in Slack,” and the firewall, the alphabet, and a stray emoji that each tried to stop us along the way.
The setup was ordinary, which is exactly why it was worth fixing. Two consumer-facing sites, three backend services shared between them, all living on one VPS, all deployed the same way: SSH in, git pull, rebuild, restart, hope. It worked, in the sense that the sites stayed up. It also meant every deploy needed a terminal, a steady hand, and nobody else’s eyes on what was about to happen until it already had.
What follows isn’t a tutorial. It’s the shape of what we actually built, and — more usefully — every place the obvious plan turned out to be wrong.
1. The shape of the pipeline
The pattern across every repository ended up the same: a build workflow that runs on every push to the main branch — tests first where there were tests, then a container image pushed to a private registry — and a completely separate deploy workflow that only ever runs when someone explicitly triggers it.
That separation was deliberate from day one. Building on every push gives you a permanent, tested, ready-to-ship artifact sitting in the registry at all times. Deploying only on request means nobody wakes up to a surprise release because a typo-fix commit happened to land at 2am. The two halves don’t know about each other beyond one thing: the image tag.
The registry choice came with its own small lesson: both the container registry and the private package feed we used for sharing code between services turned out to accept only classic access tokens, not the newer scoped ones — confirmed the hard way, after assuming otherwise once and getting corrected. Worth checking platform docs per feature, not generalizing from one that happened to work.
2. A gate the database actually needs
The backend services carry real databases, which means some deploys carry schema migrations, which means some deploys can go wrong in a way that git revert doesn’t fix. That needed its own gate, separate from the ordinary deploy path — and it needed to fail in the safe direction.
The rule we landed on: every deploy quietly diffs the commit it’s about to ship against whatever’s currently live, looking specifically at migration files. If the current deploy can’t even resolve what’s live — no marker, bad git state, anything ambiguous — the gate treats that as a migration until proven otherwise, not the reverse. Guessing wrong in the cautious direction costs a confirmation click. Guessing wrong in the other direction costs data.
Blocked — schema change pending Confirmed — backup taken Watching — health check
When a migration is genuinely coming, the deploy stops and asks for an explicit confirmation before touching anything — and once confirmed, it takes a full database backup first, verifies the backup file actually has bytes in it (an empty backup file is worse than no backup file, because it looks like safety), and only then runs the migration. Three separate real bugs got caught building this, in three separate review passes: a way to inject arbitrary commands through the confirmation input, a version of the gate that silently defaulted to “safe” on inputs it couldn’t parse, and a completely unrelated path — a project-wide “deploy everything” script — that could still reach the same containers without ever passing through the gate at all. All three only turned up because each pass asked “how would this fail,” not “does this work.”
3. When the firewall said no
The deploy workflows ran on the platform’s own hosted runners, which meant every deploy step was, under the hood, a GitHub-owned machine SSHing into the VPS. That worked for about five minutes before it didn’t — the VPS’s own firewall restricted that port to one known IP address, and a hosted runner’s IP is neither known nor stable.
The tempting fix is opening the port to the world and trusting the SSH key alone. The actual fix was smaller and more boring: install a lightweight runner agent directly on the VPS. Instead of GitHub’s cloud reaching in, the VPS itself reaches out, polls for work, and runs the deploy steps locally — no SSH, no open port, no inbound connection to defend at all. The firewall stayed exactly as restrictive as it already was.
It’s a direction-of-connection fix more than a security-tooling fix, and it’s the kind of thing that looks obvious in hindsight and invisible in advance.
4. Teaching Slack to pull the lever
The actual goal: react to a build notification with a checkmark, or reply “ship it,” and the deploy happens — no tab-switching to a CI dashboard, no hunting for the right workflow button. A small always-on bridge service does the listening, using a persistent app-to-chat connection rather than a public webhook endpoint, so it needed no domain, no TLS certificate, nothing exposed to the internet at all — just an outbound connection to Slack and a token scoped to trigger exactly one thing: the deploy workflow, nothing else.
Two bugs surfaced the moment this met a real chat client instead of a test script, and neither would have shown up in local testing with hand-typed strings:
- The emoji wasn’t the emoji. A build notification’s checkmark displays as an emoji in the client, but the chat platform’s own API hands it back to a listening bot as a text shortcode, not the character. Every message-matching rule anchored on the literal emoji silently failed — confirmed by logging exactly what the bridge received, not by guessing. The fix was matching on the surrounding sentence instead of the symbol.
- Any reply at all counted as “deploy.” The first working version triggered on literally any message in the thread — a typo, a stray “lol,” anything. That’s an easy way to ship by accident. The fix narrowed it to exactly two deliberate signals: react with the checkmark, or reply with a specific phrase. Nothing else does anything.
The bridge itself stays stateless between messages — it re-reads whatever notification you reacted to each time, rather than remembering anything — which means replying to an old message works exactly the same as replying to a brand new one, and there’s nothing to get out of sync.
5. The site that forgot how to fall back
Once both sites were ready to leave their preview path and live at their real address, they needed a safety net: if the live container is ever unreachable — mid-deploy, mid-crash — traffic should fall back to a simple static page automatically, with zero chance of a visitor ever landing on a broken page. Not a coin-flip between the two; the real site gets every request as long as it’s healthy, full stop, and the fallback only ever engages on an actual failure.
The reverse proxy handles this natively: a health check on the real service, and an error-triggered fallback that only fires on an actual server error — not a load-balanced split between two backends, which would show the fallback to some fraction of visitors even when everything’s fine. Verified against the proxy’s own documentation before trusting it, since “probably works like that” isn’t good enough for a production routing decision.
The bug that actually shipped first had nothing to do with that logic. The proxy’s file-based config loads every file in its watched directory, not just the one meant to be “live” — and the two template variants (normal mode, and a manual maintenance-mode override) were sitting in that same directory as reference copies. The proxy loaded all three, hit a naming collision, and silently kept whichever file happened to sort first alphabetically — which was the maintenance-mode template. The real site deployed successfully, health-checked green, and the fallback page kept showing anyway, because the live config had already lost a silent tie-break before any of that mattered.
Diagnosed from the proxy’s own admin API and logs, not guesswork — it logs exactly this as “already configured, skipping” when two files claim the same name. The fix was moving the templates out of the directory the proxy actually scans, so there’s only ever one real file for it to find.
6. What it actually bought
None of this is exotic. It’s the unglamorous middle of software — build, gate, deploy, fall back, notify — done carefully enough that each piece can be trusted without someone watching it happen. The measure that matters isn’t the tooling, it’s what a deploy looks like now: a checkmark on a phone, in a Slack thread, from wherever you happen to be standing. The terminal is still there if you want it. Nobody needs it anymore.
Service names, hostnames, and credentials throughout are illustrative, not real.
