JobCannon
All skills

Grafana & Prometheus

Metrics visualization & alerting: dashboards, queries, monitoring

⬢ TIER 2Tools
High
Salary impact
5 months
Time to learn
Medium
Difficulty
3
Careers
TL;DR

Grafana & Prometheus is the de facto observability stack for DevOps and SRE: Prometheus collects time-series metrics (CPU, memory, requests, latencies) from targets; Grafana visualizes them in customizable dashboards with alerting rules. Career path: Practitioner (basic dashboards, PromQL queries, $95-130k) → Architect (custom metrics, Thanos federation, multi-cluster, $140-190k) → Expert (cardinality tuning, recording rules, complex alerting rules, $180-250k) over 4-6 months. Ecosystem: Prometheus, Grafana, Loki (logs), Tempo (traces), Mimir (long-term storage), Thanos, Pyroscope, Alertmanager. Free (Prometheus) + paid (Grafana Cloud). Industry standard in 2026 — Datadog/New Relic alternatives cost 10x more at scale.

What is Grafana & Prometheus

Grafana and Prometheus form the industry standard open-source observability stack: Prometheus scrapes and stores time-series metrics (CPU, memory, requests, latencies); Grafana visualizes them in customizable dashboards with alerting rules. Career progression: Practitioner (basic dashboards, PromQL queries, $95-130k) → Architect (custom metrics, Thanos federation, multi-cluster monitoring, $140-190k) → Expert (cardinality tuning, recording rules, complex alerting, $180-250k+) over 4-6 months. Cost: self-hosted = infrastructure only, enabling unlimited scale; Grafana Cloud (managed) ≈ $0.10-2.50/GB/month. In 2026, Prometheus + Grafana is the de facto standard in tech companies—Kubernetes-native, open-source, and 10x cheaper than Datadog/New Relic at scale. Teams already using Datadog often run Prometheus in parallel for cost control. The ecosystem extends beyond the duo: Loki (logs), Tempo (traces), Mimir (long-term storage), Thanos (multi-cluster federation), Pyroscope (continuous profiling), and Alertmanager (alert routing). Learning the stack is an investment that compounds across multiple platforms.

🔧 TOOLS & ECOSYSTEM
PrometheusGrafanaAlertmanagerLokiTempoMimirPyroscopeThanosVictoriaMetricskube-state-metricsnode_exporterOpenTelemetryGrafana OnCall

📋 Before you start

Kubernetes DockerDevops Ci Cd

💰 Salary by region

RegionJuniorMidSenior
USA$98k$155k$215k
UK£55k£92k£140k
EU€62k€98k€150k
CANADAC$102kC$160kC$220k

❓ FAQ

Prometheus vs Datadog vs New Relic — why choose Prometheus?
Prometheus is open-source, free-tier unlimited (self-hosted), and has zero per-metric costs. Datadog and New Relic charge per metric (typically $0.10-1.50/metric/month). At 1,000 metrics × 12 months, Datadog costs $1,200-18,000/year; Prometheus costs $0/year (you pay for compute). Trade-off: you manage infrastructure (HA, backups, storage). For startups and cost-conscious orgs: Prometheus + Grafana. For fully-managed SaaS: Datadog/New Relic. Hybrid: Prometheus for internal, Datadog for complex third-party integrations.
What is PromQL and how difficult is it to learn?
PromQL is Prometheus's time-series query language. Learning curve: 1-2 weeks for basics (rate(), increase(), topk()). It's simpler than SQL and more intuitive than Splunk SPL. Key concepts: selectors (label matching), range vectors (5m avg), aggregation (sum/avg/max), and math. Most Prometheus users write ~30 standard queries repeatedly; advanced features (histograms, quantiles) take 4-8 weeks to master. Beginner PromQL queries: `rate(http_requests_total[5m])` (requests per second), `node_memory_MemFree_bytes / node_memory_MemTotal_bytes` (memory utilization %). Practice on public demo at play.grafana.org.
Cardinality explosion — what is it and how do I avoid it?
Cardinality = unique label combinations. If you have a metric `http_requests_total` with labels `method`, `status`, `path`, and path is unbounded (includes user IDs, timestamps, random values), you'll create millions of unique time series. Each costs memory and query time. Example: `http_requests_total{method='GET',status='200',path='/users/123'}` × thousands of user IDs = millions of series. Fix: cardinality limits in scrape config (max 10k series per target), remove unbounded labels (drop user_id), or bucket high-cardinality values (path='/users/*' instead of individual paths). Monitor with `prometheus_tsdb_metric_chunks_created_total`.
Recording rules vs alerting rules — what's the difference?
Recording rules pre-compute expensive queries and store results as new metrics. Example: instead of calculating `rate(http_requests_total[5m])` on every dashboard load, create a recording rule that computes it every 30 seconds and stores as `http_requests_per_second`. Use when: complex queries, high cardinality, multi-step math. Alerting rules fire alerts when conditions are true. Example: `alert if http_error_rate > 5%`. Recording rules are for optimization; alerting rules are for notification. Most setups use both: recording rules precompute metrics, alerting rules trigger on them.
RED vs USE method — which monitoring approach should I use?
RED (Rate-Errors-Duration) for services: monitor request rate, error rate, request duration (p50/p95/p99). Example: HTTP service, track requests/sec, 5xx error %, and response time latency. USE (Utilization-Saturation-Errors) for resources: monitor CPU/memory/disk utilization, saturation (queue depth), errors. Example: database server, track CPU %, memory %, query queue depth, failed queries. Most teams use both: RED for application-level SLOs, USE for infrastructure health. Prometheus shines at both — combine them for 360-degree observability.
Mimir vs Thanos — which long-term storage solution do I need?
Thanos is an open-source sidecar that adds 10+ years of retention and cross-cluster queries. Mimir is a fully-managed Grafana Cloud product (or self-hosted) optimized for scale and multi-tenancy. Thanos: free, DIY ops, S3-compatible storage (Minio, GCS, Azure), downsampling. Mimir: managed simplicity, native TSDB, built-in multi-tenancy, query federation, ~$0.26-2.50/GB/month for Grafana Cloud. Choose Thanos for: self-hosted, long history, cost control. Choose Mimir for: SaaS simplicity, high availability, multi-team isolation.
How do I get started — install locally or use Grafana Cloud?
For learning: install Prometheus + Grafana locally with Docker (`docker run prom/prometheus`) in 5 minutes. For production: Kubernetes Helm charts (kube-prometheus-stack = Prometheus + Grafana + Alertmanager in one chart). For managed SaaS: Grafana Cloud (free tier: 10k metrics, 7-day retention). Kubernetes teams always use kube-prometheus-stack + node_exporter + kube-state-metrics for cluster-wide observability. Most Grafana dashboards are shared at grafana.com/dashboards — import templates rather than building from scratch.

Not sure this skill is for you?

Take a 10-min Career Match — we'll suggest the right tracks.

Find my best-fit skills →

Find your ideal career path

Skill-based matching across 2,536 careers. Free, ~10 minutes.

Take Career Match — free →