Data Engineering beginner ⏱ 6–8 hours
Apache Airflow: Pipeline Orchestration
Get started with Apache Airflow running in Docker — write your first DAGs, understand scheduling, and orchestrate a real data pipeline.
AirflowOrchestrationDockerDAGs
View project on GitHub
What you’ll build
Airflow running locally with Docker Compose, with DAGs that cover the most common patterns: file sensors, Python operators, HTTP hooks, task dependencies, and retries. The orchestration foundation every data engineer needs.
Skills you’ll practice
- Airflow architecture: scheduler, webserver, workers, metadata DB
- Writing DAGs: operators, sensors, task dependencies
- XComs, connections, and variables
- Triggering and monitoring pipeline runs in the Airflow UI
The full project guide is for members
Join Surfalytics to unlock every step of this project, all other projects, the full course, and the private community.
7-day free trial · Cancel any time