https://indrajith.me/tags/apache-iceberg/ Recent content in Apache-Iceberg on Indrajith Indraprastham https://indrajith.me/dp.jpg https://indrajith.me/dp.jpg Hugo -- 0.125.5 en-us Fri, 05 Jun 2026 10:00:00 +0530 https://indrajith.me/posts/distributed-dedup-redis-bloom-filter-2-billion-records/ Fri, 05 Jun 2026 10:00:00 +0530 https://indrajith.me/posts/distributed-dedup-redis-bloom-filter-2-billion-records/ A major North American retailer needed to process two billion point-of-sale files. The existing architecture couldn’t survive two thousand. This is the story of the rebuild, the two dead ends we hit along the way, and the one design decision that made the whole thing scale. The short version: the original pipeline wasn’t slow because the code was bad. It was slow because the architecture was the wrong shape for the problem.