kestra

ELT for AEMET weather data.

Built a complete end-to-end data platform to ingest, process, and analyze complex, multi-source public datasets for business intelligence.

The Data Engineering Zoomcamp covers essential skills in containerization, workflow orchestration, data warehousing, analytics engineering, batch, and streaming processing. It includes tools like Docker, Terraform, BigQuery, dbt, Spark, Kafka, Kestra, Postgres, Google Data Studio, and Metabase.

libraries used in Kestra Flows

Kestra Auth-Bypass Vulnerability Checker

End-to-end crypto data engineering and MLOps platform with GCP, BigQuery, dbt, Kestra, Terraform, CI/CD, Looker Studio dashboards, and conservative ML signal research.

🛠️ My homework solutions & end-to-end projects for the Data Engineering Zoomcamp 2026 (DataTalks.Club) — Docker, BigQuery, dbt, Spark, Flink, Kestra & more

Multi-Agent System for Autonomous, Verifiable Knowledge Synthesis

Cloud-native batch pipeline analyzing global tech ecosystem health by correlating layoffs trends with YC startup activity. Built with Terraform, Kestra, GCS, BigQuery, dbt, and Looker Studio.

FoodMind AI is an open-source food knowledge platform integrating Wikidata, USDA FoodData Central, and Open Food Facts into Elasticsearch. It provides scalable ingestion with dlt and Kestra, typed data models with Pydantic, and agentic AI workflows built with PydanticAI for nutrition-aware search and insights.

End-to-end data engineering pipeline integrating 20 years of Kenyan climate and agricultural market data using Python, Kestra, GCS, BigQuery, and dbt to analyze how extreme weather impacts commodity prices and food supply chains.

End-to-end data engineering platform on Citi Bike data: batch (Spark), streaming (Kafka), ingestion (dlt), orchestration (Kestra), dbt, Bruin, Terraform - fully local on Docker.

Text-to-SQL agent grounded in a governed YAML semantic layer over a DuckDB star schema. Ablation on 10 metric questions: retrieval raised execution accuracy 0.50→1.00 (gpt-4o-mini) and 0.30→0.80 (local Llama 3). Hybrid retrieval + reranking, read-only SQL guardrails, 50-question eval set. FastAPI · React · Kestra · dlt · Docker.

AutoGenesis is an AI agent that turns ideas into full software projects - with code, tests, CI/CD, Docker, live preview, self‑review, and XP‑based learning.

Data Engineering ZoomCamp

Real-time phishing infrastructure detector from Certificate Transparency logs (DE Zoomcamp 2026 capstone)

Open Source software code for use with PCIe card-based hardware AI accelerators catering to both inference and training use cases

📊 End-to-end telecom data analytics and ETL pipeline using Kestra orchestration, PostgreSQL, and Python.

OpenClassrooms P10 Data Engineer - Industrialisation pipeline donnees vins BottleNeck (Kestra + DuckDB + Python)

A self-hosted, lightweight ETL pipeline for orchestrated exports from REST API data to S3 with zero infrastructure overhead. Intelligent API batching, S3 optimization, retry logic, automatic S3 gap backfilling, customizable logging.