· intermediate
Orchestration with Airflow
Running pipelines on a schedule and knowing what happened. DAGs and tasks, what the execution date really means, why a task must be safe to run twice, and how to backfill without making it worse.
Written in house, from the work rather than from a syllabus. Every example is one we would use ourselves, and every quiz question explains why its answer is right.
What you’ll learn
- Describe a pipeline as a DAG of tasks with explicit dependencies
- Explain the data interval, and why the run date is not today
- Write tasks that are safe to run twice on the same interval
- Configure retries and alerts so a failure is noticed
- Run a backfill deliberately, and avoid a stampede
- Say when a scheduler is the wrong answer
Before you start
- · Python basics
- · Data Engineering Foundations, or familiarity with ETL