System Monitoring & Observability

You can explain the difference between monitoring and observability and identify which of the three pillars solves a given production failure.

1. Why monitoring and observability matter for reliable systems2 hrs
   • uptime SLA calculations
   • mttr vs mtbf metrics
   • silent failures
   • business impact of outages
2. Monitoring vs observability: what each really means2 hrs
   • known unknowns vs unknown unknowns
   • cardinality challenges
   • white-box vs black-box telemetry
   • synthetic probing
3. The three pillars: metrics, logs and traces2 hrs
   • timeseries data structures
   • unstructured vs structured log formats
   • distributed context propagation headers
   • telemetry overhead trade-offs
4. Linux, processes, networking and systems essentials for monitoring2 hrs
   • /proc filesystem inspection
   • netstat and ss socket diagnostics
   • vmstat memory pressure indicators
   • systemd journalctl querying
5. The modern observability landscape and tooling overview2 hrs
   • Prometheus ecosystem
   • Elastic Stack vs Loki
   • OpenTelemetry standards
   • commercial SaaS pricing models

Hands-on project: Produce an architecture document and gap analysis for a legacy 3-tier web service, mapping its failure modes against metrics, logs, and traces.

You can deploy Prometheus, write complex PromQL queries, and scrape application metrics using custom exporters.

1. Time-series data and metric types: counter, gauge, histogram, summary4 hrs
   • monotonic counters
   • gauges for point-in-time values
   • histogram bucket distribution
   • quantiles in summaries
2. Prometheus architecture and the data model3 hrs
   • pull-based scraping model
   • label indexing strategies
   • TSDB head block and chunk persistence
   • remote write storage adapters
3. Installing, configuring and scraping with Prometheus3 hrs
   • prometheus.yml global config
   • static target definition
   • scrape timeouts
   • systemd service configuration
4. Exporters: node_exporter, blackbox and writing custom exporters3 hrs
   • node_exporter hardware metrics
   • blackbox_exporter HTTP probing
   • client_golang library instrumentation
   • custom metric exposition format
5. PromQL: querying, functions and aggregation4 hrs
   • rate() and irate() functions
   • histogram_quantile() calculations
   • vector matching operators
   • subquery syntax
6. Service discovery and relabeling3 hrs
   • file_sd_config dynamic targets
   • Kubernetes pod discovery
   • relabel_configs drop and keep actions
   • label sanitization rules

Hands-on project: Configure a Prometheus server in a Docker container to scrape a custom Node.js application and a node_exporter daemon, enforcing a 15-second scrape interval and custom retention rules.

You can build production-ready Grafana dashboards with template variables, annotations, and multi-source panels.

1. Grafana setup and connecting data sources4 hrs
   • Grafana server deployment
   • Prometheus data source authentication
   • provisioning datasources via YAML
   • HTTP proxy settings
2. Building dashboards, panels and visualizations4 hrs
   • time series panel configurations
   • stat and bar gauge panels
   • table transformations
   • thresholds and color mapping
3. Templating, variables and annotations4 hrs
   • query variable creation
   • custom multi-value variables
   • dashboard link annotations
   • deploy marker annotations via API
4. Visualizing Prometheus metrics with PromQL4 hrs
   • optimizing panel query performance
   • step size calculations
   • combining multiple metrics in one panel
   • transforming panel outputs
5. Dashboard design best practices and sharing4 hrs
   • layout hierarchy for incident triage
   • exporting dashboards as JSON
   • RBAC and team permissions
   • anonymous viewing configurations

Hands-on project: Build an executive Grafana dashboard connected to Prometheus that visualizes CPU saturation, memory leaks, request latency percentiles, and database connection pool exhaustion using dynamic templated dropdowns.

You can centralize, parse, and query high-volume application logs using Elasticsearch, Logstash, Kibana, and Grafana Loki.

1. Centralized logging fundamentals4 hrs
   • syslog protocol
   • stdout/stderr container log redirection
   • log parsing vs indexing trade-offs
   • storage sizing formulas
2. The Elastic Stack: Elasticsearch, Logstash and Kibana4 hrs
   • Elasticsearch cluster architecture
   • shard allocation and index lifecycle management
   • Logstash filter grok patterns
   • Kibana discover and dashboard tools
3. Log shippers: Filebeat and the Beats family4 hrs
   • Filebeat harvester configuration
   • registry file offset tracking
   • multiline log event aggregation
   • backpressure handling
4. Grafana Loki and Promtail4 hrs
   • index-free log aggregation model
   • Promtail client deployment
   • chunk storage backend configuration
   • label indexing efficiency
5. Structured logging, parsing and querying at scale4 hrs
   • JSON log formatting standards
   • LogQL stream selectors and filter expressions
   • LogQL parser stage expressions
   • regex extraction techniques

Hands-on project: Deploy a Grafana Loki and Promtail pipeline alongside Logstash to ingest JSON access logs, parse nested error fields, and construct saved searches in Kibana and Loki LogQL streams.

You can instrument microservices with OpenTelemetry, propagate trace contexts across network boundaries, and analyze bottlenecks in Jaeger and Tempo.

1. Why tracing matters in microservices4 hrs
   • black box service call failures
   • latency attribution across networks
   • synchronous vs asynchronous request flows
   • distributed debugging limits
2. Spans, traces and context propagation4 hrs
   • W3C traceparent header specification
   • span lifecycle and parent-child relationships
   • baggage propagation across threads
   • sampling rate strategies
3. OpenTelemetry: instrumentation, SDKs and the Collector4 hrs
   • auto-instrumentation agents
   • manual span creation in code
   • OTel collector receivers and exporters
   • processor batching and memory limiters
4. Jaeger and Grafana Tempo4 hrs
   • Jaeger backend ingestion architecture
   • Tempo object storage integration
   • span dependency graph generation
   • latency distribution histograms
5. Correlating metrics, logs and traces4 hrs
   • injecting trace IDs into log output
   • jumping from Grafana logs to Jaeger traces
   • exemplars in Prometheus metrics
   • unified observability workflows

Hands-on project: Instrument a multi-service Python and Go application using OpenTelemetry SDKs, configure the OpenTelemetry Collector to batch spans, and trace a distributed transaction end-to-end in Jaeger.

You can configure Alertmanager routing trees, write symptoms-based alert rules, and integrate notification channels to minimize alert fatigue.

1. Designing meaningful alerts: symptoms vs causes2 hrs
   • user-impacting symptom alerts
   • resource exhaustion warning alerts
   • avoiding actionable noise
   • paging criteria definitions
2. Prometheus Alertmanager: rules, routing and silencing2 hrs
   • alert recording and alerting rules syntax
   • routing tree matchers and groupings
   • silence creation via API and CLI
   • repeat intervals and resolution delays
3. Notification channels: Slack, email, PagerDuty and Opsgenie2 hrs
   • webhook receiver configurations
   • PagerDuty API integration keys
   • Slack attachment templates
   • email digest configurations
4. On-call, escalation policies and runbooks2 hrs
   • escalation chain design
   • primary and secondary on-call rotations
   • runbook documentation standards
   • handoff procedures
5. Reducing alert fatigue2 hrs
   • alert auditing and review cycles
   • flapping alert mitigation
   • aggregation groups
   • tolerating transient errors

Hands-on project: Build a Prometheus alerting rule set with Alertmanager routing configuration that sends high-severity pages to PagerDuty and warning alerts to a Slack channel, incorporating inhibition rules to silence cascading alarms.

You can apply the RED and USE methodologies to profile application performance and track throughput, latency, and error rates.

1. APM concepts and the RED and USE methods2 hrs
   • Rate, Errors, Duration framework
   • Utilization, Saturation, Errors framework
   • choosing metrics by resource layer
   • bottleneck isolation
2. Instrumenting applications for metrics and traces2 hrs
   • runtime profiling hooks
   • garbage collection telemetry
   • thread pool saturation monitoring
   • database connection pool metrics
3. Tracking latency, throughput and error rates2 hrs
   • p95 and p99 latency tracking
   • requests per second calculation
   • HTTP status code error partitioning
   • payload size histograms
4. APM tools: Datadog, New Relic and Elastic APM2 hrs
   • Datadog agent deployment
   • service map auto-discovery
   • error tracking and stack trace grouping
   • profiling CPU bottlenecks
5. Real user and synthetic monitoring overview2 hrs
   • browser performance metrics (Core Web Vitals)
   • synthetic transaction scripts
   • global edge probing
   • uptime SLA reporting

Hands-on project: Instrument a production web service with Datadog APM, capture custom database query spans, and set up an anomaly detection monitor for HTTP latency spikes.

You can monitor host infrastructure and cloud services across AWS, Azure, and GCP using native and open-source tooling.

1. Host and infrastructure metrics2 hrs
   • CPU iowait and context switching
   • disk inode and free space exhaustion
   • virtual memory paging
   • network interface drops
2. Cloud-native monitoring: AWS CloudWatch, Azure Monitor and GCP Operations2 hrs
   • CloudWatch metrics and metric math
   • Azure Monitor diagnostic settings
   • GCP Cloud Monitoring dashboards
   • cloud audit log ingestion
3. Uptime, synthetic and endpoint monitoring2 hrs
   • TCP and HTTP endpoint checks
   • SSL certificate expiration tracking
   • DNS resolution monitoring
   • geo-distributed probing
4. Network and database monitoring2 hrs
   • PostgreSQL pg_stat_statements monitoring
   • Redis memory fragmentation ratios
   • packet loss and latency metrics
   • firewall drop counters
5. Capacity and cost monitoring2 hrs
   • resource utilization forecasting
   • cloud cost allocation tags
   • right-sizing recommendations
   • storage growth trends

Hands-on project: Configure AWS CloudWatch alarms and metric streams to export EC2 and RDS performance data into a centralized Prometheus and Grafana monitoring instance.

You can deploy the kube-prometheus-stack to monitor pods, nodes, persistent volumes, and Kubernetes control plane components.

1. Monitoring containers with cAdvisor4 hrs
   • cAdvisor container metrics extraction
   • cgroup resource limits and throttling
   • container memory working set metrics
   • ephemeral storage usage
2. kube-state-metrics and cluster health4 hrs
   • deployment rollout status metrics
   • pod restart and crash loop tracking
   • persistent volume claim binding states
   • node readiness conditions
3. The Prometheus Operator and the kube-prometheus-stack4 hrs
   • Custom Resource Definitions (CRDs)
   • ServiceMonitor and PodMonitor objects
   • Helm chart customization values
   • operator lifecycle management
4. Monitoring pods, nodes, workloads and the control plane4 hrs
   • kube-apiserver latency monitoring
   • etcd leader election and disk sync duration
   • kube-scheduler scheduling queue latency
   • kubelet health endpoints
5. Logging and tracing inside Kubernetes4 hrs
   • FluentBit DaemonSet log collection
   • OpenTelemetry operator for Kubernetes
   • sidecar injection patterns
   • namespace log isolation

Hands-on project: Deploy the Prometheus Operator via Helm onto a multi-node Kubernetes cluster, configuring ServiceMonitors for custom applications and establishing resource quota alerts for namespaces.

You can define Service Level Indicators, SLOs, and error budgets, and conduct blameless postmortems after incidents.

1. Site Reliability Engineering principles2 hrs
   • embracing risk
   • managing toil limits (50% rule)
   • service ownership models
   • automation vs manual intervention
2. SLIs, SLOs, SLAs and error budgets2 hrs
   • selecting valid Service Level Indicators
   • setting realistic Service Level Objectives
   • calculating error budget depletion
   • consequences of budget exhaustion
3. Defining and measuring reliability2 hrs
   • multi-window multi-burn-rate alerts
   • window-based vs request-based SLOs
   • reporting reliability to stakeholders
   • reliability reviews
4. Blameless postmortems and incident reviews2 hrs
   • cultural foundations of blamelessness
   • timeline reconstruction techniques
   • root cause vs contributing factors
   • tracking preventive action items
5. Capacity planning and reducing toil2 hrs
   • load testing correlation with metrics
   • identifying repetitive operational tasks
   • automating remediation runbooks
   • capacity horizon modeling

Hands-on project: Write an SLO definition document for a core authentication microservice, calculate burn rate alert thresholds, and author a blameless postmortem template complete with action items.

You can forecast capacity from your own metrics and stop an observability bill from outgrowing the service.

1. Forecasting from your own metrics90 min
   • Trend and seasonality
   • Headroom and saturation
   • Lead time for provisioning
2. Cardinality is the cost driver90 min
   • Label explosion
   • Finding top series
   • Dropping and relabelling
3. Retention and downsampling90 min
   • Recording rules
   • Long-term storage tiers
   • Thanos or Mimir basics
4. Sampling logs and traces90 min
   • Head vs tail sampling
   • Keeping the errors
   • Budget per service

Hands-on project: Use historical Prometheus data to forecast when a service exhausts headroom, then cut observability spend: drop high-cardinality labels, set per-tier retention and downsampling, and report the reduction in series count and storage.

You can detect an outage before a user reports it, and prove the system survives a failure you chose.

1. Synthetic and real-user monitoring90 min
   • Black-box probes
   • Critical journey checks
   • RUM vs synthetic
2. Designing a chaos experiment90 min
   • Steady-state hypothesis
   • Blast radius and abort criteria
   • Game-day format
3. Injecting realistic failure90 min
   • Pod and node kill
   • Latency and packet loss
   • Dependency outage
4. Closing the loop90 min
   • Did monitoring catch it
   • Fixing what broke
   • Regression as a scheduled run

Hands-on project: Add synthetic checks for the critical user journey from two regions with alerting on failure, then run a controlled chaos experiment that kills a dependency, and document the hypothesis, blast radius and what the experiment revealed.

You can design, deploy, and operate a complete production-grade monitoring, logging, and tracing stack with defined SLOs and incident runbooks.

1. Design an end-to-end monitoring and observability stack4 hrs
   • architecture diagramming for telemetry
   • capacity planning for observability backends
   • security and access control planning
   • integration boundary definition
2. Deploy Prometheus, Grafana, logging and tracing4 hrs
   • automated deployment via Helm and GitOps
   • storage volume provisioning and retention
   • network policy hardening
   • tls certificate configuration
3. Configure dashboards, alerts and SLOs4 hrs
   • unified operational dashboard creation
   • Alertmanager routing validation
   • SLO burn rate alert testing
   • synthetic probe verification
4. Write runbooks and an incident-response plan4 hrs
   • authoring triage runbooks linked to alerts
   • simulating failure scenarios (chaos testing)
   • incident commander role protocols
   • postmortem simulation exercise
5. Portfolio, GitHub, mock interviews and career prep4 hrs
   • GitHub repository structuring and documentation
   • SRE and DevOps interview question patterns
   • troubleshooting scenario walkthroughs
   • resume positioning for observability roles

Hands-on project: Deploy an end-to-end observability stack on Kubernetes containing Prometheus, Grafana, Loki, and OpenTelemetry, instrumented against a microservices application, configured with PagerDuty alerts, and documented with an operational runbook for portfolio review.