Observability is the property of a system that allows you to understand its internal state by examining its external outputs. The distinction between observability and monitoring is important: monitoring is the practice of watching known metrics and alerting when they cross thresholds. Observability is the practice of building systems that can answer arbitrary questions about their internal state, including questions you didn’t think to ask when you built the system. In practice, monitoring tells you that something is wrong. Observability tells you why.
The three pillars of observability are metrics, logs, and traces, and understanding how they work together is more important than understanding any one of them in isolation.
Metrics are numerical time-series data: the count of requests per second, the 99th percentile response latency, the CPU usage of a server, the number of database connections in use. Metrics are efficient to collect and store, easy to graph and trend, and ideal for alerting (you set thresholds and are notified when they are crossed). Their limitation is that they show you what is happening in aggregate but not why it is happening for any specific case.
Logs are immutable records of discrete events. Every time something meaningful happens in your system (a request is received, a database query is executed, an error is thrown), your application should emit a log record describing what happened. The critical distinction here is between structured logging and unstructured logging. Unstructured log lines like “Error processing request for user 12345” are difficult to aggregate, filter, and analyze at scale. A structured log record emits the same information as a machine-readable object: {"level": "error", "service": "order-processor", "user_id": 12345, "error": "payment_timeout", "downstream": "payment-service", "duration_ms": 5000}. Structured logs can be queried, filtered, and aggregated programmatically, which makes them a much more powerful debugging tool when you are investigating a production incident and need to find all requests that failed in a specific way.
Traces address the challenge specific to distributed systems: following a single request as it moves through multiple services. When a user’s checkout request touches your web server, your order service, your payment service, your inventory service, and your notification service before returning a response, a trace records the entire journey: when each service received the request, how long it spent processing, which downstream calls it made, and which of those calls was slow or failed. A trace provides the “story” of a request in a way that neither metrics nor logs alone can provide.
The SLI, SLO, and SLA hierarchy is the framework that gives your observability investments meaning and connects them to business value. A Service Level Indicator (SLI) is a specific, carefully chosen metric that represents user experience. The key word is “chosen”: your system produces thousands of metrics, but only a few of them actually reflect whether users are having a good experience. Good SLIs for most services include request success rate (the proportion of requests that complete without an error) and latency (specifically, the proportion of requests that complete within a given time threshold, because an average latency hides the experience of slow users). A Service Level Objective (SLO) is a target value for an SLI over a given time window: “99.5% of requests should succeed, measured over a rolling 30-day window.” An SLO answers the question “how reliable do we need to be?” Without it, you either over-engineer reliability (building systems that aim for five-nines availability when your users would be fine with three-nines) or under-engineer it (not discovering that users are unhappy until they’ve left). A Service Level Agreement (SLA) is a contractual commitment to an SLO, with defined consequences (financial or otherwise) for violation.
The error budget is derived from the SLO and is one of the most powerful concepts in reliability engineering. If your SLO is 99.9%, then 0.1% of requests may fail: that is your error budget. If you have consumed your error budget for the month (your failure rate has exceeded 0.1%), the engineering team should stop deploying new features and focus exclusively on reliability improvements until the system is stable again. This creates a direct, automatic alignment between reliability and feature velocity: the more reliable the system is, the faster you can deploy. The more outages you have, the more your deployment velocity is constrained. This makes reliability a self-funding priority rather than a cost center.
Alerting philosophy is often neglected but operationally critical. A well-designed alert is actionable: it fires when there is something an engineer can and should do right now, and it includes enough context (what is wrong, which service, what the likely cause is) to begin that action immediately. A poorly designed alert fires whenever a metric crosses a threshold, regardless of whether any action is warranted or even possible. Alert fatigue, the state where on-call engineers routinely ignore alerts because so many of them turn out to be noise, is one of the most dangerous failure modes in operations, because it means that the alert that matters will eventually be ignored too.