The deploy pipeline that does not page you at 3am
A good deploy pipeline is boring on purpose: small changes, fast rollbacks, and enough observability to know what broke before a customer tells you.
The best compliment a deploy pipeline can get is that nobody thinks about it. It runs, it’s fast, it rolls back cleanly, and it doesn’t wake anyone up. Getting there isn’t about exotic tooling. It’s about a handful of boring decisions made early and kept.
Small changes, often
The single biggest predictor of a calm on-call rotation is the size of your deploys. Big, infrequent releases bundle ten changes together, so when something breaks you get to bisect ten suspects at 3am.
Ship small. A change that touches one thing is a change you can reason about, roll back, and forget. “Deploy on every merge to main” sounds scary until you realise it’s the thing that makes each deploy boring.
The pipeline is just scripts you can read
We’ve inherited pipelines built from four tools stitched together with custom YAML DSLs that nobody on the team fully understood. When they broke, the team was helpless.
Our default is the opposite: CI is GitHub Actions calling shell scripts you can run on your laptop.
# deploy.sh — the whole thing fits in your head
set -euo pipefail
./scripts/build.sh
./scripts/migrate.sh # forward-only, tested, reversible
./scripts/health-check.sh # fail loudly before we shift traffic
./scripts/promote.sh # shift traffic, keep the old version warmIf a step fails, you can run that one script by hand and see exactly what happened. No magic, no proprietary debugger.
Rollback is a feature, not an incident
Ask a team “how do you roll back?” and the answer tells you everything. If it’s “we revert the commit and wait for a fresh build,” your rollback is measured in tens of minutes — during an outage.
Keep the previous version deployable and one command away:
- Immutable build artifacts tagged by commit SHA.
- The previous version kept warm so traffic can shift back instantly.
- Database migrations that are forward-only and backward-compatible, so rolling back code never strands you against a schema it can’t read.
That last point is where most rollbacks actually fail. Decouple the deploy from the migration: ship the migration first, make sure old code still works against the new schema, then ship the code.
Observability is part of the deliverable
You can’t roll back what you can’t see. Before a feature is “done,” it has:
- Structured logs with a request ID you can trace end to end.
- A few metrics that matter — error rate, latency, and a business signal (orders, signups, whatever the feature exists to produce).
- One alert that means something. Alert on symptoms users feel (error rate, latency), not causes (CPU at 80%). An alert nobody acts on is just noise training your team to ignore the next one.
This isn’t “phase two.” A feature without observability is a feature you’re flying blind on the moment it ships.
The boring setup, in order
If you’re standing up a pipeline this week:
- Deploy on merge to main, automatically.
- One readable deploy script, runnable locally.
- Immutable artifacts and a one-command rollback.
- Forward-only, backward-compatible migrations, deployed separately from code.
- Structured logs with a trace ID.
- One symptom-based alert wired to the channel your on-call actually watches.
None of this is novel. That’s the point. The reward for a boring pipeline is that the exciting part of your week is the product, not the infrastructure.