01
Context
Supplier, financial and operational information was distributed across independently managed procurement, contract, claims and payment platforms. Differences in identifiers, definitions and data quality made it difficult to establish a consistent enterprise view.
I developed Snowflake and dbt transformations that connected these datasets, identified inconsistencies and supported validation for reporting and migration work.
02
Challenge
The same supplier could appear differently across procurement and finance systems, with multiple locations, identifiers or payment relationships. Reliable reconciliation therefore required more than comparing fields with similar names: the logic had to reflect the real business relationships and distinguish genuine discrepancies from legitimate structural differences.
Large data volumes added a second constraint. Matching and reporting processes needed to remain traceable while avoiding unnecessary reprocessing and compute use.
03
Approach
I created intermediate models that standardised each source before comparing key attributes and relationships. Modular dbt models and SQL macros made common transformation patterns reusable, while incremental processing limited repeated work.
Apache Airflow supported scheduled transformations, and row- and column-level access controls protected governed datasets. I also optimised a large-scale matching and reporting pipeline operating across more than 240 million records.
04
Technical detail
- Integration across supplier, site, contract, claims and payment data
- Reconciliation logic for missing records, conflicting attributes and inconsistent relationships
- Modular dbt models and reusable SQL macros
- Incremental processing and Airflow orchestration
- Row-level and column-level access controls
- Query and transformation optimisation across more than 240 million records
05
Outcome
The resulting datasets improved visibility across supplier and financial systems, made discrepancies easier to trace and strengthened migration validation. Pipeline optimisation reduced compute usage by approximately 30% while preserving the detail needed to investigate exceptions.