Data Engineering intermediate ⏱ 12–15 hours
Data Ingestion Patterns: API to Snowflake
A 6-week deep dive into multiple data ingestion approaches using the GitHub API as a source and Snowflake as the target.
SnowflakePythonAPIIngestionAirflow
View project on GitHub
What you’ll build
Six different ingestion approaches to load the same GitHub API data into Snowflake: raw Python, Airbyte, Fivetran, custom connectors, and Airflow orchestration. By the end you understand the trade-offs behind every ingestion decision.
Skills you’ll practice
- REST API ingestion: pagination, auth, rate limits
- Multiple Snowflake load patterns: PUT + COPY, Snowpipe, Connector
- Comparing ELT tools: Airbyte vs custom vs managed connectors
- Incremental ingestion and deduplication strategies