The Monitoring You Set Up During an Outage Is the Wrong Monitoring

Every dashboard built at 2 a.m. is shaped like the outage you’re currently having.

Something is wrong in production. The site is slow, or a service is down, or a customer noticed before you did. You SSH in and start reconstructing what happened from whatever the machine still remembers: the last few thousand lines of a log file, the output of top, a disk that turns out to be at 97 percent.

You get there eventually. And somewhere in that process you have the thought everyone has, which is that this would have taken four minutes instead of two hours if you’d had any visibility at all.

So you add some. And the monitoring you build in that state is shaped precisely like the outage you just had: a dashboard for the thing that broke, an alert for the specific threshold you just crossed. It’s better than nothing. It is also almost useless against the next failure, which will be different, because failures usually are.

The problem isn’t that people don’t value observability. It’s that the moment you most want it is the moment you’re least equipped to build it well.

Why it never gets done first

Monitoring has an unusual property among infrastructure work: on the day you finish it, it produces nothing.

There’s no feature to demo. No page loads faster. No customer notices. You’ve spent a day or two and the observable result is a dashboard showing that everything is, as far as anyone knew, fine. It’s insurance, and insurance always loses the argument against work with a visible output, right up until the day it doesn’t.

Then there’s the honest counterargument, which is that you genuinely might not need it yet. A single droplet running a side project does not require Prometheus. That’s a real position, and I’d rather steelman it than pretend everyone needs a monitoring hub on day one.

But it has a failure mode, and the failure mode is that “when we need it” always arrives as an emergency. Nobody schedules the outage. So the tooling gets built under time pressure, by someone who is also trying to restore service, which is how you end up with monitoring designed around one incident. The decision isn’t really “now or later.” It’s “deliberately, or during a fire.”

What actually goes in the box

Strip the vendor language off and a small-team observability stack is four jobs, and I’ll name what I use for each, from the stack I built and open-sourced for exactly this:

Metrics: Prometheus, plus a Node Exporter on every machine. Numbers over time. CPU, memory, disk, and whatever your application chooses to expose. This is what tells you a box has been slowly filling its disk for eleven days.

Logs: Loki, with Promtail shipping from every node. The text your systems emit. Metrics tell you something broke; logs usually tell you what.

Alerts: Alertmanager. The part that reaches out and finds you, rather than waiting for you to look.

One pane of glass: Grafana. Somewhere all of it renders together, so answering a question doesn’t require SSH access to four machines.

Worth being direct about what that list doesn’t include: there’s no tracing in it. No Tempo, no Jaeger, no OpenTelemetry. Distributed tracing is genuinely valuable, and it’s also a different kind of project, one that requires instrumenting your application code rather than deploying infrastructure alongside it. For a team running a handful of services, metrics plus logs plus alerts is the large majority of the value for a small fraction of the work. Scope decisions you can defend out loud beat completeness you never finish.

The alert thresholds are the actual work

Standing up the containers is the easy half. Anyone can get Prometheus running in an afternoon. Deciding what deserves to interrupt your evening is the part that takes judgment, and it’s where most homegrown setups go wrong in one of two directions.

Too quiet, and it misses things. Too loud, and you start ignoring it, which is strictly worse than having nothing: you’ve now got an alerting system and a trained reflex to dismiss it.

Here’s the pattern that works, taken from the rules I actually ship:

- alert: HighMemoryUsage
  expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
  for: 5m
  labels:
    severity: warning

- alert: CriticalMemoryUsage
  expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 95
  for: 2m
  labels:
    severity: critical

The same metric, twice, and the important detail is that the two tiers differ in both dimensions. Not just the threshold, the patience. A box touching 91 percent memory has to hold there for five full minutes before it says anything, because brief spikes are normal and a system that pages you for normal behavior is a system you’ll mute. At 95 percent, the tolerance drops to two minutes, because now the trajectory matters more than the noise.

Severity isn’t a label you attach to an alert. It’s a claim about how long you’re willing to let the condition continue, and if those two things don’t move together you’ve built a pager that cries wolf.

Disk gets the same two-tier treatment (20 percent free is a warning, 10 percent is critical), with one small addition that matters more than it looks:

expr: (node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / ...

That filter excludes tmpfs and Docker’s overlay filesystems. Without it, a host running containers fires disk alerts constantly about filesystems that are supposed to be full and aren’t real disks. That single exclusion is the difference between an alert you act on and an alert you write a mail rule to hide.

Every noisy alerting system I’ve seen got that way one reasonable-seeming rule at a time. The discipline is to treat every alert as a promise that you’ll do something when it fires, and to delete the ones where that promise isn’t true.

The three alerts that watch the watchers

Here’s the part I’d most want a team to steal, because it’s the one people leave out.

A monitoring stack is software, and software fails. When yours fails, it usually doesn’t fail loudly. Promtail stops shipping. A scrape target quietly drops off. Alertmanager loses its connection and every alert it would have routed goes nowhere at all.

And your dashboards keep rendering. Everything looks calm, because the thing responsible for telling you otherwise is the thing that’s broken. A silently dead monitoring stack is worse than no monitoring stack, because it manufactures confidence you haven’t earned.

So three of the nine rules I ship don’t watch the infrastructure at all. They watch the monitoring:

- alert: PrometheusTargetMissing
  expr: up == 0
  for: 5m

- alert: LokiNotReceivingLogs
  expr: sum(rate(loki_distributor_lines_received_total[5m])) == 0
  for: 10m

- alert: AlertmanagerNotConnected
  expr: prometheus_notifications_alertmanagers_discovered < 1
  for: 5m

A log pipeline receiving zero lines for ten minutes isn’t quiet, it’s broken. A Prometheus that can’t discover an Alertmanager is a system that will observe your next outage in perfect detail and tell nobody.

If that instinct feels familiar, it’s the same one I’ve written about in a completely different context: a guardrail that lives inside the thing it’s guarding isn’t a guardrail. Monitoring has the identical recursive problem. Something has to be watching the watcher, and the answer isn’t infinite regress, it’s making the failure of your observability itself an observable event.

Deploy it as code, or you’ll do it once

The last piece is how it gets stood up, and this is where a monitoring stack either becomes permanent infrastructure or becomes a thing you built once and can’t reproduce.

Mine is Terraform for the infrastructure (VPC, the management node, firewall rules, a Spaces bucket for long-term log archival) and Ansible for the stack itself, running the components under Docker Compose. One terraform apply, one ansible-playbook run.

The reason that matters isn’t elegance. It’s that hand-built monitoring is monitoring you can’t rebuild, which means you can’t move it, can’t recover it, and won’t touch it once it’s working because nobody remembers exactly how it got that way. The alert thresholds above are the clearest case: as code, they’re reviewable in a pull request and diffable when someone changes them at 3 a.m. Clicked into a web UI, they’re tribal knowledge with a login page.

There’s a related benefit that sounds like a platitude and isn’t. The configuration becomes the documentation. Nobody writes a wiki page listing every alert threshold, and if they do it’s wrong within a month. The rules file is always current, because it’s the thing actually running.

For access, everything management-facing stays off the public internet and is reached over Tailscale, which I’ve written about at length elsewhere and won’t repeat here. Grafana is the single pane; Prometheus, Loki, and Alertmanager aren’t exposed even on the tailnet, since Grafana queries them over the internal Docker network. A dashboard full of your infrastructure’s weak points is not something to leave with a login form facing the open internet.

The honest version

I’ll say the quiet part, because a piece arguing for doing this early should be straight about how the incentives actually run.

The reason monitoring gets postponed isn’t ignorance. Every engineer who has skipped it knew better. It’s that observability is one of the few kinds of work whose entire value is contingent on a future bad day, and humans are famously bad at spending today’s hours on that. That’s not a knowledge problem you can lecture someone out of. It’s a scheduling problem, and the only fix I know is to treat it as part of standing up infrastructure rather than as a separate project that earns its own justification.

Not “we’ll add monitoring when we need it.” Just: a node isn’t finished until something is watching it.

The dashboard you build during an outage answers exactly one question, and you already know the answer, because you’re living it. The one you build beforehand answers the question you haven’t thought to ask yet.

That’s the whole difference, and it’s only available in advance.