"How would you know what's happening in your system? What would you monitor?" Volunteering observability shows you think past the happy path — exactly what senior interviewers look for.
⏱ 30 minPrereq: Units 08–12You'll be able to: add monitoring, SLOs, and good alerts to any design
By the end of this unit
Distinguish the three pillars: metrics, logs, traces.
Monitor the four golden signals and know what each catches.
Define SLI/SLO/SLA and use error budgets.
Write alerts that page on symptoms, not noise.
The three pillars
Metrics
Numeric time-series (QPS, latency, error rate, CPU, queue depth). Cheap to store, great for dashboards & alerts. Tells you that something's wrong.
Logs
Timestamped event records, ideally structured (JSON). Tells you what happened on a specific request. Aggregate centrally (ELK, Loki).
Traces
The path of one request across services with timing per hop. Tells you where the time/error went in a distributed call. (OpenTelemetry, Jaeger.)
Soundbite "Metrics tell me that something broke, traces tell me where, logs tell me why. I'd emit all three with a shared request id so I can pivot between them during an incident."
The four golden signals
If you only watch four things per service, watch these (from Google SRE):
Signal
Catches
Latency (esp. p95/p99)
Slowness users feel. Watch tail latency, not just the average.
Traffic (QPS)
Load level; context for everything else.
Errors (rate / %)
Failures — 5xx, exceptions, failed jobs.
Saturation
How full the system is — CPU, memory, disk, queue depth, connection pools.
Averages lie A 50 ms average can hide a p99 of 3 s — and the p99 is the experience of your most active users. Always reason in percentiles for latency.
SLIs, SLOs, SLAs & error budgets
SLI — a measured indicator (e.g., % of requests < 300 ms, % non-5xx).
SLO — your internal target for an SLI (e.g., 99.9% of requests succeed/month).
SLA — a contractual promise to customers, with penalties; usually looser than the SLO.
Error budget — the allowed unreliability (100% − SLO). 99.9% ⇒ ~43 min/month of "budget." Spend it on risk/velocity; when it's exhausted, slow down and stabilize.
Alerting that doesn't cry wolf
Good alerts are symptom-based and actionable: page a human when users are affected (error budget burning fast, p99 over SLO, queue backing up unboundedly), not on every CPU blip. Noisy alerts cause fatigue, and fatigue causes missed real incidents.
Page on user-facing symptoms; send cause-y signals to dashboards/tickets.
Use burn-rate alerts (how fast you're eating the error budget) over static thresholds.
Every page should have a runbook: what it means, what to check, how to mitigate.
Don't alert on causes Alerting "CPU > 80%" pages you even when users are fine. Alert on the symptom ("error rate > SLO" / "p99 > 1s") and use CPU as a dashboard metric to diagnose after you're paged.
Quick check
A request is slow and you need to know which of 6 microservices ate the latency. Which pillar?
Tracing. A trace stitches together the spans of one request as it crosses services, so you can see exactly which hop was slow or errored. Metrics tell you there's a problem; the trace localizes it.
Quick check
Your SLO is 99.9% successful requests/month. What does the error budget represent?
Allowed unreliability. 100% − SLO is your budget for downtime/errors. While budget remains, you can ship faster and take risks; once it's burned, the policy is to freeze risky changes and focus on reliability. (An SLA is the separate, contractual promise.)
Practice before you move on
For the photo app, list: the four golden signals you'd dashboard per service, two things you'd page on (symptom-based), one SLO you'd set, and how a shared request id ties metrics, logs, and traces together during an incident.