A traceable retail price pipeline that turns heterogeneous product pages into validated, analytics-ready price history.
The Operating Context
"Price intelligence depends on comparable historical observations, not a single scrape. This project makes the collection-to-mart path inspectable so analysts can trust where price records came from, how they were transformed, and which run produced them."
Comparable retail prices are distributed across heterogeneous product pages, where source-specific extraction, incomplete attributes, and changing page structures can quietly corrupt downstream analysis. The workflow needed to preserve what was collected, reject invalid records, and publish repeatable price history rather than one-off snapshots.
Built an Airflow-orchestrated daily batch pipeline with eight independent retailer-category branches. Each branch extracts source data, stores raw evidence and manifests in MinIO Bronze, validates and writes typed Parquet into Silver, loads run-scoped records into PostgreSQL, then fans into dbt staging and Gold marts.
A production-shaped local data platform where reruns remain traceable through run IDs, retries are explicit, rejected records stay inspectable, and price observations accumulate without overwriting history.
Airflow schedules a daily DAG with eight independent source-category branches across four retailers and two product categories.
Retailer adapters collect raw product evidence and manifests into MinIO so each ingestion run remains traceable before transformation.
Orchestration evidence: One Airflow DAG coordinates 8 independent source-category branches, then waits for each landing load before the dbt build fan-in.
Quality evidence: The repository includes 49 pytest checks and 12 dbt tests across extraction, parsing, loading, and warehouse modeling.
Reliability evidence: Run IDs, retries, rejected-record handling, and idempotent source-category loads make failures and reruns inspectable rather than silent.
Evidence before transformation: Retaining Bronze payloads and manifests makes a source change diagnosable and gives downstream tables a traceable origin.
Data contracts matter: Typed validation and a rejected-record path keep malformed retailer data from leaking into warehouse models just because a scrape completed.
Idempotency is a delivery feature: Run-scoped loads and append-only price observations let the pipeline be rerun without erasing or duplicating the history analysts need.