JobCannon
All skills

Unknown

⬢ TIER 2Tech
High
Salary impact
6 months
Time to learn
Medium
Difficulty
5
Careers
TL;DR

Apache Airflow is the industry standard workflow orchestration platform for data engineering: define pipelines as directed acyclic graphs (DAGs), schedule execution via cron or event triggers, and monitor thousands of tasks across clusters. Path: Junior Data Engineer (DAG basics, operators, XComs; 4-6 months → $95-120k) → Mid (custom operators, Kubernetes, Snowflake/BigQuery integration; 6-12 months → $120-155k) → Senior (dynamic DAGs, performance tuning, multi-tenant architecture; 18+ months → $155-190k). Runs on-premises, on Kubernetes, or managed platforms (MWAA/Astronomer/Cloud Composer). Pricing: self-hosted = infra costs; managed ≈ $1-3/DAG/day.

What is Unknown

Apache Airflow is the industry-standard workflow orchestration platform for data engineering: define complex data pipelines as Directed Acyclic Graphs (DAGs), schedule execution via cron or event triggers, and monitor thousands of tasks across clusters. A DAG is a set of tasks with explicit dependencies (Task A → Task B → Task C). Airflow ensures Task B runs only after Task A succeeds. Career progression: Junior Data Engineer (DAG basics, operators, XComs, 4-6 months → $95-120k) → Mid (custom operators, Kubernetes, warehouse integration, 6-12 months → $120-155k) → Senior (dynamic DAGs, performance tuning, multi-tenant architecture, 18+ months → $155-190k). Pricing: self-hosted = infrastructure costs; managed (MWAA/Astronomer) ≈ $1-3/DAG/day. In 2026, Airflow is the de facto standard for ETL/ELT orchestration at companies >$100M revenue; 10k+ companies (Netflix, Uber, Adobe, Twitter) use it. Startups often start with simpler tools (dbt) before graduating to Airflow. Airflow competes with Prefect (modern API, simpler) and Dagster (asset-centric, better lineage), but Airflow's ecosystem and adoption make it the safe choice. Learning Airflow takes 3-6 months; mastery (performance tuning, multi-tenant setup) takes 18+ months.

🔧 TOOLS & ECOSYSTEM
Apache Airflow CoreMWAA (AWS Managed Workflows)Astronomer PlatformGoogle Cloud ComposerCelery ExecutorKubernetes ExecutorPostgreSQL metadata DBdbtSnowflakeBigQueryApache SparkKafka

💰 Salary by region

RegionJuniorMidSenior
USA$105k$145k$185k
UK£62k£88k£125k
EU€68k€95k€135k
CANADAC$115kC$160kC$210k

❓ FAQ

Airflow vs Prefect vs Dagster — when do I use each?
Airflow: Python-native, 10k+ companies, best ecosystem, steepest DAG learning curve. Prefect: modern API, better error handling, less operational overhead, smaller community. Dagster: asset-centric (not task-centric), best for data assets/lineage, younger product. For enterprise data teams: Airflow. For rapid prototyping: Prefect. For assets-first thinking: Dagster.
Self-hosted vs managed Airflow (MWAA/Astronomer/Cloud Composer) — which should I pick?
Self-hosted: full control, low cost for small teams (just Postgres + web server on EC2), but you own patching, scaling, backups. Managed: $1-3k/month base, frees you from ops, built-in monitoring, auto-scaling. For <10 DAGs or learning: self-hosted on a tiny EC2. For >50 DAGs or mission-critical: managed platform.
What's the difference between Celery, Kubernetes, and LocalExecutor?
LocalExecutor: single-machine parallelism, good for dev/testing, max ~8 workers. Celery: distributed task queue, scales to 100s of workers, requires Redis/RabbitMQ, complex ops. Kubernetes: modern, scales elastically, requires K8s cluster, best for cloud-native shops. Start LocalExecutor, graduate to Celery for on-prem, or K8s for cloud.
How do I handle dynamic DAGs (DAGs that vary per tenant or config)?
DAGFactory: generate DAGs from YAML configs at parse time. TaskFlow API decorators: cleaner than legacy operators. DAG factories iterate over configs, yielding DAG objects. Common pattern: loop over 100 customer configs, yield 100 DAGs at runtime. Avoid hardcoding: DAGs should read from a database, S3, or config file.
Why is my Airflow scheduler slow / DAGs parsing slowly?
Typical culprits: (1) too many imports in DAG file (move to operators/hooks), (2) Postgres metadata DB is overloaded (add indexes, tune max_connections), (3) too many DAGs (>5k) — split into separate Airflow instances. Solution: profile DAG parse time with `airflow dags list --verbose`, offload heavy logic to operators (not DAG definition).
How do I monitor Airflow in production?
Built-in: Airflow web UI (task/DAG status, logs, gantt). CloudWatch (if MWAA): logs → CloudWatch. Add: Datadog/New Relic for APM, PagerDuty for alerting. Set alarms on: DAG failure rate, task duration p99, scheduler lag, Postgres connection pool. Log all DAG runs + task logs to S3.
What's the difference between XComs, variables, and connections?
Variables: static config (e.g., API keys) — set in UI or code, shared across DAGs. Connections: credentials (username + password + host), used by hooks/operators. XComs: task-to-task messaging (e.g., task A computes a path, task B reads it) — ephemeral, stored in metadata DB. For secrets: use Secrets Backend (AWS Secrets Manager, Vault) instead of plain variables.

Not sure this skill is for you?

Take a 10-min Career Match — we'll suggest the right tracks.

Find my best-fit skills →

Find your ideal career path

Skill-based matching across 2,536 careers. Free, ~10 minutes.

Take Career Match — free →