Observability & SRE

Why This Matters in 2026

Observability stopped being a dashboard-installation task years ago, and by 2026 that distinction is what separates a junior monitoring setup from a real SRE practice. OpenTelemetry has become the vendor-neutral standard for instrumenting traces, metrics, and logs, which means the differentiator is no longer “which APM did you wire up” but how deliberately you designed signal coverage and SLOs before an incident ever happened. SLO and error-budget thinking is now the shared language teams use to define “what quality means here” in concrete, negotiable terms instead of vague uptime promises. At the same time, AIOps-assisted anomaly detection is increasingly used to cut through alert noise and shorten root-cause analysis, but it only works on top of well-structured signals — it doesn’t replace the design work. The strongest SRE profiles design for reliability up front; they don’t retrofit dashboards after the third 3 a.m. page.

Core Skills & Tools

• OpenTelemetry instrumentation spanning traces, metrics, and logs, with consistent semantic conventions across services
• Defining SLIs/SLOs and writing an error-budget policy that has real teeth (gates releases, triggers freezes, reallocates priorities)
• Dashboard and APM design with Grafana, Prometheus, Datadog, or New Relic, built around user-facing signals rather than infrastructure vanity metrics
• Alert design that separates actionable, page-worthy signals from informational noise (multi-window, multi-burn-rate alerting on SLOs)
• AIOps-assisted anomaly detection and cross-signal correlation (logs + metrics + traces) to accelerate root-cause analysis
• Building runbooks and dashboards that are actually used during an incident, not just referenced in a postmortem template

What You Must Have Operated

• Defined and operated SLOs backed by an error-budget policy that was actually enforced — for example, it blocked or slowed a risky release rather than existing only on paper
• Led an alert-noise-reduction initiative with a measurable before/after, not just a one-time pruning of a few rules
• Built and used a cross-signal (logs + metrics + traces) debugging workflow during a real production incident, not a tabletop exercise
• Rolled out OpenTelemetry instrumentation across multiple services and made the resulting data the default source of truth for on-call engineers

Evidence You Can Show

KPIs & Metrics

• Mean Time to Detect (MTTD) — how quickly a real degradation is identified after it starts
• Alert noise reduction (%) — drop in non-actionable alerts after a tuning or redesign effort
• SLO attainment rate — percentage of measurement windows where the SLO target was met
• Instrumentation coverage (%) — share of services/critical paths with OpenTelemetry traces, metrics, and logs wired in
• Supporting metrics: error-budget burn rate, time-to-correlate-signals during an incident, on-call pages per engineer per week

Maturity Levels

Proof Statements You Can Use

• “Reduced mean time to detect (MTTD) for critical service degradations from 22 minutes to 4 minutes by rolling out OpenTelemetry-based distributed tracing.”
• “Cut non-actionable alert volume by 68% through multi-burn-rate SLO alerting, lowering on-call pages per engineer from 14/week to 4/week.”
• “Defined and enforced an error-budget policy that blocked 3 risky releases in one quarter, holding SLO attainment above 99.5% for the core checkout path.”
• “Drove OpenTelemetry instrumentation coverage from 35% to 92% of production services, making trace-based root-cause analysis the default incident workflow.”

Field Cases

Real incident write-ups from ICT Problem Notes that exercise this competency:

• A 40-minute outage passed with no alert, and the error-rate panel showed a flat line
• All dashboards developed gaps and Prometheus restarted every twenty minutes
• p99 latency spikes to 400ms while CPU utilisation sits at 30%
• Every query got slower during business hours and recovered overnight, for three weeks

The mechanisms behind them — Absent data is not zero · Metric cardinality

Hands-On Labs

Step-by-step labs from handson — install guides, runbooks and playbooks that were actually run, not just read:

• Prometheus — instrumenting a service, and three queries that lie if you read them wrong
• Grafana — wiring metrics, logs and traces together so one ID walks between them
• Loki — logs you can query, and the one label that multiplies your streams by 25
• OpenTelemetry tracing — one trace across two services, and where the 420ms actually went
• OpenSearch — a single node, and the range query that quietly lies
• Topic of the day — Prometheus 3.13 LTS, a self-scrape lab and the new min_of()/max_of()
• SLOs and error budgets — burn-rate alerts that page once, and the SLI that records nothing

Checklists

Run-before-you-ship lists from IT Checklists — use them as the review gate behind the evidence above:

• Observability
• Production Readiness Review

Automation Case Studies

Industry patterns and incident write-ups from IT Automation that show this competency at work across organizations:

• Error Budgets and the SRE Model
• The Alert That Never Fired

Reference Library

Background chapters, ready-to-fill documents and pipeline walkthroughs from the sibling sites — see the full Reference Library for everything else:

Architecture Field Notes — Observability: What to Actually Wire · SLOs, Error Budgets, and Saying No · Capacity, Load Shedding, and Backpressure

Automation Playbook — Alert Design and Noise Reduction · Automating Observability Instrumentation