Surfalytics
All pet projects
Data Engineering beginner ⏱ 6–8 hours

Apache Airflow: Pipeline Orchestration

Get started with Apache Airflow running in Docker — write your first DAGs, understand scheduling, and orchestrate a real data pipeline.

AirflowOrchestrationDockerDAGs
View project on GitHub

What you’ll build

Airflow running locally with Docker Compose, with DAGs that cover the most common patterns: file sensors, Python operators, HTTP hooks, task dependencies, and retries. The orchestration foundation every data engineer needs.

Skills you’ll practice

  • Airflow architecture: scheduler, webserver, workers, metadata DB
  • Writing DAGs: operators, sensors, task dependencies
  • XComs, connections, and variables
  • Triggering and monitoring pipeline runs in the Airflow UI