Posts for: #Testing

Test Parallelization and Flaky Test Detection

2026-07-29sohilladhani

A test suite with 10,000 tests takes 2 hours to run sequentially. Distributed across 50 machines, it takes 2-3 minutes. Test parallelization is a solved problem in principle but creates real distributed systems challenges: test isolation, work distribution, and flaky tests that fail intermittently and poison the signal. Work Distribution Naively split 10,000 tests evenly: 200 per machine. The problem: tests have wildly varying durations. 200 fast unit tests complete in 10 seconds.

[Read more]

Testing Eventually Consistent Systems: When Assertions Need Patience

2026-03-26sohilladhani

#distributed-systems #system-design #testing #consistency #architecture

You write a record, immediately read it back, and assert equality. The test fails. Not because of a bug, but because the read hit a replica that hasn’t caught up yet. Your test is correct. Your assertion timing isn’t.

[Read more]

Contract Testing: Verifying Service Interactions Without E2E Tests

2026-03-25sohilladhani

#distributed-systems #system-design #testing #microservices #architecture

Team A changes their API response. Team B’s service breaks in production. The integration test suite passed because it was running against a mock from 3 months ago.

[Read more]

Chaos Engineering: Breaking Things on Purpose

2026-03-24sohilladhani

#distributed-systems #system-design #testing #resilience #architecture

Your system passed all tests. Every health check is green. You’re confident it handles failures. Then a network partition happens in production and everything falls apart. You never actually tested failure.

[Read more]