translation cache

Uber Rewrites Translation Cache to Cut Memory 90% While Keeping Latency Under 1ms

A 90% reduction in memory footprint while keeping p99 miss latency under 100 microseconds is the kind of engineering that makes you stop and look twice.

3 min readInfoQ
Uber Rewrites Translation Cache to Cut Memory 90% While Keeping Latency Under 1ms

Uber's decision to redesign its Go-based translation cache is the kind of engineering story that deserves more attention than it usually gets. A 90% reduction in memory footprint, a cache hit rate above 99.9%, and p99 disk-backed miss latency under 100 microseconds, these numbers are not incremental. They represent a fundamental rethinking of how a critical piece of infrastructure should behave. What Uber achieved here is a reminder that the most impactful innovation often happens not in flashy new products but in the invisible systems that make those products possible.

The hybrid architecture Uber built, keeping keys and frequently accessed values in memory while pushing the rest to disk, sounds simple in retrospect. That is the hallmark of good engineering. By reducing garbage-collection overhead and dramatically lowering memory consumption, Uber did not just make its translation cache cheaper to run. It made it more predictable. For any organization operating at scale, predictability is the difference between a system that hums along and one that burns through resources during peak load. This matters because cache performance directly affects user experience. A ride request, a delivery ETA, a driver match, each of these interactions depends on fast, reliable data retrieval. Uber's work shows that you can have both speed and efficiency without compromise. It is worth asking how many other legacy systems in your own stack are overdue for a similar rethink.

This focus on infrastructure efficiency stands in interesting contrast to Uber's more visible bets. The company is simultaneously testing a program that allows Uber pilots a new program allowing children as young as 10 to ride alone in three U.S. cities. That initiative pushes the boundaries of trust and safety. Meanwhile, Uber and Zipline are aiming for Drones and doorsteps converge as Uber and Zipline aim for a million daily deliveries by 2029. Both of these efforts demand a backend that can handle more volume, more complexity, and more real-time decision-making. A translation cache that uses 90% less memory while keeping latency under a millisecond is not a footnote to those ambitions. It is a foundation for them. Without infrastructure that scales efficiently, ambitious product expansions become expensive experiments. Uber's cache redesign suggests the company understands that the path to growth runs through the data center as much as through the app store.

The concrete takeaway here is direct: if your organization relies on in-memory caches for performance, you should examine your memory-to-value ratio. Uber proved that a hybrid disk-backed approach can cut memory consumption by an order of magnitude without sacrificing speed. That is not a theoretical insight. It is a practical design pattern that any team using Go or similar languages could evaluate today. The question is not whether this is possible, Uber has demonstrated that it is. The question is whether your own systems are carrying the same hidden overhead, and how much you could save by looking at them with fresh eyes.

From InfoQ

Uber redesigned its Go-based translation cache to reduce its initial memory footprint by more than 90% while maintaining a cache hit rate above 99.9% and p99 disk-backed cache miss latency below 100 microseconds. The hybrid architecture combines in-memory keys and frequently accessed values with disk-backed storage, reducing garbage-collection overhead and memory consumption.

Read the original at InfoQ