End-to-end retail lakehouse on Databricks: dual-path ingestion from PostgreSQL and S3, dbt-modeled galaxy schema with SCD2 history, Airflow orchestration with parallel DAG execution, CI-gated dbt tests, and a dual consumption layer via Power BI and a Genie Agent.
Incremental batch data pipeline for e-commerce order data, built with Airflow and Polars using a medallion (bronze/silver/gold) architecture, Pandera data quality gates, and dimensional modeling on DuckDB.
End-to-End Retail ETL Pipeline using Apache Airflow, PostgreSQL, Docker, and Python with Incremental Loading and Data Quality Validation.
Building my first ETL pipelines using Python to learn more about data engineering. Learning by doing.
End-to-end ETL pipeline: incremental API ingestion, dbt transforms, Airflow orchestration, and CI
Production-inspired ETL pipeline built with Python and PostgreSQL featuring data validation, transformation, incremental loading, audit logging, reporting, archiving, CLI support, retry mechanism, and automated testing.
A Databricks data engineering project simulating a full medallion architecture (Bronze → Silver → Gold) for e-commerce sales data, featuring incremental loading, a Kimball-style star schema, and SCD Type 1 & Type 2 implementations using PySpark and Delta Lake MERGE INTO.
This project pulls historical and forecast weather data for multiple cities, cleans and transforms it, performs quality checks, and stores the results in tidy daily and monthly summary datasets.
Midterm Data Pipeline using Python, PySpark, MongoDB, Data Quality, Upsert and Incremental Loading.
Real-time AWS ecommerce data pipeline using MSK Serverless, MSK Connect, S3, AWS Glue, Delta Lake, SCD Type 2, and Redshift Serverless.
Azure SQL → Azure Data Factory → ADLS Gen2 → Databricks: watermark-based incremental loads, Auto Loader, Delta Live Tables with SCD Type 2. Guided project, extended.
Local-first NYC TLC lakehouse with Bronze/Silver/Gold layers, data quality, incremental loading, and Airflow backfills
Built a metadata-driven Azure lakehouse that incrementally ingests Spotify-style SQL data with ADF, processes new Parquet files with Databricks Auto Loader, and publishes SCD-managed Delta facts and dimensions.
Real-time AWS ecommerce data pipeline using MSK Serverless, MSK Connect, S3, AWS Glue, Delta Lake, SCD Type 2, and Redshift Serverless.
End-to-end Retail Sales ETL Pipeline using Python, PostgreSQL, Apache Airflow, Docker, Star Schema Data Warehouse Modeling, KPI Generation, and Incremental Data Loading.
An end-to-end retail data ETL pipeline that extracts CSV data, transforms and validates it with Pandas, and incrementally loads it into PostgreSQL. Includes automated testing, structured logging, idempotent processing, CPU-optimized loading, and HashiCorp Vault for secure credential management.
Production-ready ETL Pipeline that extracts GitHub repositories, performs incremental loading into PostgreSQL, sends email notifications, and runs automatically using APScheduler.
End-to-End Data Engineering Pipeline using Snowflake, dbt, AWS, and Medallion Architecture (Bronze, Silver, Gold) with Incremental Models, Snapshots, Macros, and Data Quality Testing.
Fibbie Banks is a fictional retail financial institution created as a comprehensive case study to design, build, and implement a modern, production‑grade data warehouse solution. This project demonstrates end‑to‑end data engineering best practices.
End-to-end data engineering and business intelligence platform built using Databricks, PySpark, SQL, Delta Lake, and Power BI with incremental ETL pipelines and Medallion Architecture.
To associate your repository with the incremental-loading topic, visit your repo's landing page and select "manage topics."