https://sohilladhani.com/blog/tags/stream-processing/Recent content in Stream-Processing on Sohil Ladhani BlogHugoen-usSat, 04 Apr 2026 00:00:00 +0000https://sohilladhani.com/blog/post/2026-04-04-lambda-and-kappa-architecture/Sat, 04 Apr 2026 00:00:00 +0000https://sohilladhani.com/blog/post/2026-04-04-lambda-and-kappa-architecture/Real-time results are fast and approximate. Historical results are slow and accurate. The tension between them is where Lambda and Kappa architecture come from. Lambda: Two Pipelines Lambda runs two parallel systems. The batch layer processes all historical data on a schedule (Spark on HDFS, every few hours) and produces ground truth. The speed layer processes the live stream (Kafka Streams or Flink) for low-latency results. The serving layer merges both: “latest batch result plus stream delta since the last batch.https://sohilladhani.com/blog/post/2026-04-03-watermarks-and-late-data/Fri, 03 Apr 2026 00:00:00 +0000https://sohilladhani.com/blog/post/2026-04-03-watermarks-and-late-data/There are two clocks in any stream processing system. Event time: when the click actually happened, recorded in the payload. Processing time: when your system received it. On a healthy network they’re close. In reality they’re not. Mobile clients buffer events when offline. Retries add delay. A click at 10:00:05 might reach your processor at 10:00:47. The 10:00 window has long since closed. The Problem With Never Waiting If you never close a window, you never produce output.https://sohilladhani.com/blog/post/2026-04-02-stream-processing-windows/Thu, 02 Apr 2026 00:00:00 +0000https://sohilladhani.com/blog/post/2026-04-02-stream-processing-windows/Aggregating over an infinite stream sounds easy until you realize you have no idea when it ends. You need to cut it into chunks. That’s what windows are. Three Window Types Tumbling windows are fixed, non-overlapping buckets. “Clicks per minute” is a tumbling window: minute 1, minute 2, minute 3, no overlap. Simple to implement, but events that span the boundary get split across buckets. Sliding windows overlap. “Average clicks in the last 5 minutes, recomputed every minute” means each event can appear in up to 5 windows.