JobCannon
全部技能

Grafana & Prometheus

Metrics visualization & alerting: dashboards, queries, monitoring

⬢ 層級 2工具
高
薪資影響
5 個月
學習時長
中等
難度
3
職業
一覽

Grafana & Prometheus is the de facto observability stack for DevOps and SRE: Prometheus collects time-series metrics (CPU, memory, requests, latencies) from targets; Grafana visualizes them in customizable dashboards with alerting rules. Career path: Practitioner (basic dashboards, PromQL queries, $95-130k) → Architect (custom metrics, Thanos federation, multi-cluster, $140-190k) → Expert (cardinality tuning, recording rules, complex alerting rules, $180-250k). Ecosystem: Prometheus, Grafana, Loki (logs), Tempo (traces), Mimir (long-term storage), Thanos, Pyroscope, Alertmanager. Free (Prometheus) + paid (Grafana Cloud). Industry standard in 2026, Datadog/New Relic alternatives cost 10x more at scale.

什麼是 Grafana & Prometheus

Grafana and Prometheus form the industry standard open-source observability stack: Prometheus scrapes and stores time-series metrics (CPU, memory, requests, latencies); Grafana visualizes them in customizable dashboards with alerting rules. Career progression: Practitioner (basic dashboards, PromQL queries, $95-130k) → Architect (custom metrics, Thanos federation, multi-cluster monitoring, $140-190k) → Expert (cardinality tuning, recording rules, complex alerting, $180-250k+) over 4-6 months. Cost: self-hosted = infrastructure only, enabling unlimited scale; Grafana Cloud (managed) ≈ $0.10-2.50/GB/month. In 2026, Prometheus + Grafana is the de facto standard in tech companies, Kubernetes-native, open-source, and 10x cheaper than Datadog/New Relic at scale. Teams already using Datadog often run Prometheus in parallel for cost control. The ecosystem extends beyond the duo: Loki (logs), Tempo (traces), Mimir (long-term storage), Thanos (multi-cluster federation), Pyroscope (continuous profiling), and Alertmanager (alert routing). Learning the stack is an investment that compounds across multiple platforms.

🔧 工具與生態
PrometheusGrafanaAlertmanagerLokiTempoMimirPyroscopeThanosVictoriaMetricskube-state-metricsnode_exporterOpenTelemetryGrafana OnCall

💰 各地區薪資

地區初級中級高級
USA$98,000$155,000$215,000
UK£55,000£92,000£140,000
EU€62,000€98,000€150,000
CANADACA$102,000CA$160,000CA$220,000

🎯 使用 Grafana & Prometheus 的職業

Devops Engineer
Observability Engineer
Site Reliability Engineer

❓ 常見問題

Prometheus vs Datadog vs New Relic, why choose Prometheus?
Prometheus is open-source, free-tier unlimited (self-hosted), and has zero per-metric costs. Datadog and New Relic charge per metric (typically $0.10-1.50/metric/month). At 1,000 metrics × 12 months, Datadog costs $1,200-18,000/year; Prometheus costs $0/year (you pay for compute). Trade-off: you manage infrastructure (HA, backups, storage). For startups and cost-conscious orgs: Prometheus + Grafana. For fully-managed SaaS: Datadog/New Relic. Hybrid: Prometheus for internal, Datadog for complex third-party integrations.
What is PromQL and how difficult is it to learn?
PromQL is Prometheus's time-series query language. Learning curve: 1-2 weeks for basics (rate(), increase(), topk()). It's simpler than SQL and more intuitive than Splunk SPL. Key concepts: selectors (label matching), range vectors (5m avg), aggregation (sum/avg/max), and math. Most Prometheus users write ~30 standard queries repeatedly; advanced features (histograms, quantiles) take 4-8 weeks to master. Beginner PromQL queries: `rate(http_requests_total[5m])` (requests per second), `node_memory_MemFree_bytes / node_memory_MemTotal_bytes` (memory utilization %). Practice on public demo at play.grafana.org.
Cardinality explosion, what is it and how do I avoid it?
Cardinality = unique label combinations. If you have a metric `http_requests_total` with labels `method`, `status`, `path`, and path is unbounded (includes user IDs, timestamps, random values), you'll create millions of unique time series. Each costs memory and query time. Example: `http_requests_total{method='GET',status='200',path='/users/123'}` × thousands of user IDs = millions of series. Fix: cardinality limits in scrape config (max 10k series per target), remove unbounded labels (drop user_id), or bucket high-cardinality values (path='/users/*' instead of individual paths). Monitor with `prometheus_tsdb_metric_chunks_created_total`.
Recording rules vs alerting rules, what's the difference?
Recording rules pre-compute expensive queries and store results as new metrics. Example: instead of calculating `rate(http_requests_total[5m])` on every dashboard load, create a recording rule that computes it every 30 seconds and stores as `http_requests_per_second`. Use when: complex queries, high cardinality, multi-step math. Alerting rules fire alerts when conditions are true. Example: `alert if http_error_rate > 5%`. Recording rules are for optimization; alerting rules are for notification. Most setups use both: recording rules precompute metrics, alerting rules trigger on them.
RED vs USE method, which monitoring approach should I use?
RED (Rate-Errors-Duration) for services: monitor request rate, error rate, request duration (p50/p95/p99). Example: HTTP service, track requests/sec, 5xx error %, and response time latency. USE (Utilization-Saturation-Errors) for resources: monitor CPU/memory/disk utilization, saturation (queue depth), errors. Example: database server, track CPU %, memory %, query queue depth, failed queries. Most teams use both: RED for application-level SLOs, USE for infrastructure health. Prometheus shines at both, combine them for 360-degree observability.
Mimir vs Thanos, which long-term storage solution do I need?
Thanos is an open-source sidecar that adds 10+ years of retention and cross-cluster queries. Mimir is a fully-managed Grafana Cloud product (or self-hosted) optimized for scale and multi-tenancy. Thanos: free, DIY ops, S3-compatible storage (Minio, GCS, Azure), downsampling. Mimir: managed simplicity, native TSDB, built-in multi-tenancy, query federation, ~$0.26-2.50/GB/month for Grafana Cloud. Choose Thanos for: self-hosted, long history, cost control. Choose Mimir for: SaaS simplicity, high availability, multi-team isolation.
How do I get started, install locally or use Grafana Cloud?
For learning: install Prometheus + Grafana locally with Docker (`docker run prom/prometheus`) in 5 minutes. For production: Kubernetes Helm charts (kube-prometheus-stack = Prometheus + Grafana + Alertmanager in one chart). For managed SaaS: Grafana Cloud (free tier: 10k metrics, 7-day retention). Kubernetes teams always use kube-prometheus-stack + node_exporter + kube-state-metrics for cluster-wide observability. Most Grafana dashboards are shared at grafana.com/dashboards, import templates rather than building from scratch.

不確定這項技能適不適合你？

做一下職業匹配，我們會推薦合適的方向。

找到最適合我的技能 →

找到理想的職業方向

在 2,521 個職業中進行技能匹配。免費。

免費做職業匹配 →