Introduction to Module 4

Data integrations and ETL: from SQL and BI to your first data engineering work.

Where we are

Module 4 is about data integration and ETL. So far in the course:

Module 1–3: what business intelligence is, who a BI developer and a data analyst are, and the key skills: databases, BI tools, SQL.
Prerequisites: Git and the command line.

Now you learn the core skill of data warehousing: moving data from different sources into a database. Both data warehouses and integrations are old ideas that have been around for a very long time.

This is the first step toward a data engineer role, and it covers almost 50% of the work of an analytics engineer.

What you will learn

ETL: Extract, Transform, Load.
ELT: Extract, Load, Transform, and how it differs from ETL.
Hands-on practice with ETL tools on the Superstore dataset.
Dimensional modeling in practice: in Module 2 you learned what data modeling is. Here you build your first slowly changing dimension model with ETL jobs and pipelines.

Tools in this module

Tool	What it is	Why we use it
Pentaho Data Integration	Open source ETL tool with a visual UI. Works on Windows, Linux and macOS	We start here and build pipelines on the Superstore dataset
dbt	Transformation tool	The “T” in modern ELT
Airflow	Open source orchestration	Scheduling and running pipelines
Meltano and similar	Open source and commercial cloud offerings	A look at the cloud side of integration
PostgreSQL	Database	The target where we load data

Maybe we will also spin up a cloud database and practice there.

How to work through this module

Learn the ETL vs ELT difference first. It shapes everything else.
Install Pentaho Data Integration and rebuild the pipelines from the lessons on the Superstore dataset.
Load the result into PostgreSQL and apply the dimensional modeling skills from Module 2.

Next steps

See all lessons in Module 4: Data Integration and ETL.
Build an end-to-end pipeline as a pet project.
Compare the data engineer path with other roles in Getting your first data job.