https://sohilladhani.com/blog/tags/data-engineering/Recent content in Data-Engineering on Sohil Ladhani BlogHugoen-usSat, 20 Jun 2026 00:00:00 +0000https://sohilladhani.com/blog/post/2026-06-20-backfill-and-reprocessing/Sat, 20 Jun 2026 00:00:00 +0000https://sohilladhani.com/blog/post/2026-06-20-backfill-and-reprocessing/You add a new derived field to your user profiles: “days since last purchase.” You can compute it for new events going forward. But existing users already have purchase history. You need to compute this field for all existing users retroactively. That’s a backfill. Backfills happen constantly in production systems. They’re more disruptive than they look. The Basic Problem You have a dataset of 500 million user records. You need to reprocess all of them to compute the new field.