Monitoring vs Observability
Monitoring tells you when something is wrong. Observability tells you why. While monitoring checks known failure modes against predefined thresholds, observability gives you the tools to investigate unknown failure modes by examining the internal state of your system through its external outputs.
Monitoring: The Foundation
Monitoring is about collecting predefined metrics and alerting when they cross thresholds. CPU usage above 90%, error rate above 1%, response time above 500ms — these are monitoring checks. Monitoring works well for known failure modes but struggles with novel problems. "The dashboard shows everything is green, but users are complaining" is the classic monitoring failure.
Observability: Understanding the Unknown
Observability is a property of a system — the degree to which you can understand its internal state from its external outputs. An observable system lets you ask arbitrary questions without deploying new instrumentation. It emerged from complex distributed systems where the number of possible failure modes is too large to predefine.
The Three Pillars: Metrics
Metrics are numerical measurements collected over time: request rate, error rate, latency percentiles, CPU usage, memory consumption. They are cheap to store, fast to query, and ideal for dashboards and alerting. Counter, gauge, histogram, and summary are the four metric types in Prometheus. Metrics answer "what" is happening. Use the Timestamp Converter to work with metric timestamps and time ranges.
The Three Pillars: Logs
Logs are timestamped, immutable records of discrete events: "User 42 logged in", "Payment processing failed: timeout", "Deployment v2.3.1 started." They provide rich context for individual events. Structured logging (JSON format) makes logs queryable and parseable. The challenge is volume — a busy service can generate gigabytes of logs per hour. Format and validate structured log output with the JSON Formatter.
The Three Pillars: Traces
Distributed traces follow a single request as it flows through multiple services. A trace consists of spans — each span represents one operation (an API call, a database query, a cache lookup) with timing, status, and metadata. Traces answer "where" time was spent and "which" service failed. OpenTelemetry is the emerging standard for instrumentation. Without traces, debugging a slow request in a microservices architecture is like finding a needle in a haystack.
SLOs, SLIs, and Error Budgets
A Service Level Indicator (SLI) is a quantitative measure of service quality (e.g., "99.2% of requests complete in under 200ms"). A Service Level Objective (SLO) is the target for that SLI (e.g., "99.9% of requests in under 200ms over a 30-day window"). The error budget is the allowed failure: 100% minus the SLO target. SLO-based alerting is more actionable than threshold alerting — it tells you whether you are burning through your error budget, not just whether a metric crossed a line.
Alerting Strategy
Good alerts are actionable, directed at the right person, and fire only when human intervention is needed. Alert on symptoms (user-facing error rate), not causes (CPU usage) — high CPU that does not affect users should not page someone at 3 AM. Use multiple urgency levels: pages for critical user impact, tickets for degradation, and dashboards for informational metrics. Use the Crontab Calculator to schedule regular health check scripts that feed into your monitoring system.
Tooling Landscape
Metrics: Prometheus + Grafana, Datadog, CloudWatch. Logs: ELK Stack (Elasticsearch, Logstash, Kibana), Loki, Splunk. Traces: Jaeger, Zipkin, Honeycomb, Datadog APM. OpenTelemetry is unifying instrumentation across all three pillars. For small teams, a managed solution (Datadog, Grafana Cloud) avoids operational overhead. For cost-sensitive environments, the open-source stack (Prometheus + Loki + Tempo + Grafana) is powerful but requires expertise to run. Format your configuration files with the JSON to YAML Converter — most observability tools use YAML configuration.