JobCannon
全部技能

Apache Airflow

⬢ 層級 2技術
高
薪資影響
6 個月
學習時長
中等
難度
5
職業
一覽

Apache Airflow is the industry standard workflow orchestration platform for data engineering: define pipelines as directed acyclic graphs (DAGs), schedule execution via cron or event triggers, and monitor thousands of tasks across clusters. Path: Junior Data Engineer (DAG basics, operators, XComs; 4-6 months → $95-120k) → Mid (custom operators, Kubernetes, Snowflake/BigQuery integration; 6-12 months → $120-155k) → Senior (dynamic DAGs, performance tuning, multi-tenant architecture; 18+ months → $155-190k). Runs on-premises, on Kubernetes, or managed platforms (MWAA/Astronomer/Cloud Composer). Pricing: self-hosted = infra costs; managed ≈ $1-3/DAG/day.

什麼是 Apache Airflow

Apache Airflow is the industry-standard workflow orchestration platform for data engineering: define complex data pipelines as Directed Acyclic Graphs (DAGs), schedule execution via cron or event triggers, and monitor thousands of tasks across clusters. A DAG is a set of tasks with explicit dependencies (Task A → Task B → Task C). Airflow ensures Task B runs only after Task A succeeds. Career progression: Junior Data Engineer (DAG basics, operators, XComs, 4-6 months → $95-120k) → Mid (custom operators, Kubernetes, warehouse integration, 6-12 months → $120-155k) → Senior (dynamic DAGs, performance tuning, multi-tenant architecture, 18+ months → $155-190k). Pricing: self-hosted = infrastructure costs; managed (MWAA/Astronomer) ≈ $1-3/DAG/day. In 2026, Airflow is the de facto standard for ETL/ELT orchestration at companies >$100M revenue; 10k+ companies (Netflix, Uber, Adobe, Twitter) use it. Startups often start with simpler tools (dbt) before graduating to Airflow. Airflow competes with Prefect (modern API, simpler) and Dagster (asset-centric, better lineage), but Airflow's ecosystem and adoption make it the safe choice. Learning Airflow takes 3-6 months; mastery (performance tuning, multi-tenant setup) takes 18+ months.

🔧 工具與生態
Apache Airflow CoreMWAA (AWS Managed Workflows)Astronomer PlatformGoogle Cloud ComposerCelery ExecutorKubernetes ExecutorPostgreSQL metadata DBdbtSnowflakeBigQueryApache SparkKafka

💰 各地區薪資

地區初級中級高級
USA$105,000$145,000$185,000
UK£62,000£88,000£125,000
EU€68,000€95,000€135,000
CANADACA$115,000CA$160,000CA$210,000

❓ 常見問題

Airflow vs Prefect vs Dagster, when do I use each?
Airflow: Python-native, 10k+ companies, best ecosystem, steepest DAG learning curve. Prefect: modern API, better error handling, less operational overhead, smaller community. Dagster: asset-centric (not task-centric), best for data assets/lineage, younger product. For enterprise data teams: Airflow. For rapid prototyping: Prefect. For assets-first thinking: Dagster.
Self-hosted vs managed Airflow (MWAA/Astronomer/Cloud Composer), which should I pick?
Self-hosted: full control, low cost for small teams (just Postgres + web server on EC2), but you own patching, scaling, backups. Managed: $1-3k/month base, frees you from ops, built-in monitoring, auto-scaling. For <10 DAGs or learning: self-hosted on a tiny EC2. For >50 DAGs or mission-critical: managed platform.
What's the difference between Celery, Kubernetes, and LocalExecutor?
LocalExecutor: single-machine parallelism, good for dev/testing, max ~8 workers. Celery: distributed task queue, scales to 100s of workers, requires Redis/RabbitMQ, complex ops. Kubernetes: modern, scales elastically, requires K8s cluster, best for cloud-native shops. Start LocalExecutor, graduate to Celery for on-prem, or K8s for cloud.
How do I handle dynamic DAGs (DAGs that vary per tenant or config)?
DAGFactory: generate DAGs from YAML configs at parse time. TaskFlow API decorators: cleaner than legacy operators. DAG factories iterate over configs, yielding DAG objects. Common pattern: loop over 100 customer configs, yield 100 DAGs at runtime. Avoid hardcoding: DAGs should read from a database, S3, or config file.
Why is my Airflow scheduler slow / DAGs parsing slowly?
Typical culprits: (1) too many imports in DAG file (move to operators/hooks), (2) Postgres metadata DB is overloaded (add indexes, tune max_connections), (3) too many DAGs (>5k), split into separate Airflow instances. Solution: profile DAG parse time with `airflow dags list --verbose`, offload heavy logic to operators (not DAG definition).
How do I monitor Airflow in production?
Built-in: Airflow web UI (task/DAG status, logs, gantt). CloudWatch (if MWAA): logs → CloudWatch. Add: Datadog/New Relic for APM, PagerDuty for alerting. Set alarms on: DAG failure rate, task duration p99, scheduler lag, Postgres connection pool. Log all DAG runs + task logs to S3.
What's the difference between XComs, variables, and connections?
Variables: static config (e.g., API keys), set in UI or code, shared across DAGs. Connections: credentials (username + password + host), used by hooks/operators. XComs: task-to-task messaging (e.g., task A computes a path, task B reads it), ephemeral, stored in metadata DB. For secrets: use Secrets Backend (AWS Secrets Manager, Vault) instead of plain variables.

不確定這項技能適不適合你？

做一下職業匹配，我們會推薦合適的方向。

找到最適合我的技能 →

找到理想的職業方向

在 2,521 個職業中進行技能匹配。免費。

免費做職業匹配 →