Case study · Insurance
Rebuilding the pipeline for a motor insurer
An ETL rebuild that moved 1M+ records a day from 15+ sources without choking.
Delivery work from our track record. We've left the client unnamed; the numbers are real.
60%
faster ingestion
90%
faster nested transforms
1M+
records a day
The challenge
The old pipeline couldn't keep up. Nested data and growing volume meant every overnight load ran late, and every report ran later.
What we did
- Re-architected ingestion end to end, for both historical loads and daily increments.
- Wrote a dynamic flattener for the messy nested JSON that was slowing everything down.
- Tuned compute so 15+ sources ran on a budget instead of brute force.
Stack
AzureApache SparkDatabricksDelta Lake