Cache-testing software for LLM-provider-style tiered ephemeral caches? [D]

When a Reddit user recently posted a detailed request for cache-testing software designed specifically for LLM-provider-style tiered ephemeral caches, it surfaced a quiet but important gap in the AI infrastructure stack. The poster had already tried libCacheSim, a well-regarded classical cache simulator, and found it fundamentally mismatched for the problem at hand. The issue was not a lack of effort but a lack of tooling built for a world where AI systems manage context across multiple tiers with different residency windows, costs, and eviction rules. This is the kind of infrastructure challenge that rarely makes headlines but quietly shapes whether AI services feel responsive or frustrating — and it connects directly to the kinds of real-world AI deployments we are seeing across the industry. Google's recent move to add Gemini-powered Dictation to Gboard, for instance, depends on exactly the kind of low-latency, context-aware caching infrastructure that tools like the one being sought would help evaluate. Similarly, Waymo's software recall to address flooding conditions and TikTok's evolving transaction layer both rely on backend systems where intelligent caching of real-time data can determine whether an AI-driven decision arrives in milliseconds or too late to matter.

What makes this gap particularly revealing is the specificity of the requirements. Classical cache simulators were built for single, flat structures — think CPU caches or CDN edge nodes where objects are uniform in size and misses carry roughly equivalent costs. LLM prompt caching inverts that model entirely. A single conversation might span four or more cache tiers, with partial hits at different levels carrying dramatically different cost profiles. A prompt cached in the fastest tier costs nearly nothing to retrieve, while a full recompute from scratch represents the most expensive miss. On top of that, cached objects are not static — they mutate in place as conversations evolve, and token-weighted sizing means that a 500-token snippet and a 5,000-token snippet consume resources unevenly. No mainstream benchmark suite models this behavior natively, which means teams building these systems are essentially flying blind when it comes to evaluating eviction policies at scale.

This is not a niche problem waiting for a niche solution. As more AI products move into production — handling longer contexts, serving more concurrent users, and operating under tighter cost constraints — the demand for rigorous, purpose-built simulation tooling will only grow. Research prototypes and internal tools at major AI labs likely address fragments of this challenge, but nothing has emerged as a shared, community-driven standard. The closest analogs are KV-cache simulators used in inference serving, which model token generation overhead but typically ignore multi-tier residency and edit-driven workloads. What the community needs, and what does not yet exist in an accessible form, is a framework that treats the cache hierarchy as a first-class architectural concern — one where cost gradients, object mutability, and token-aware sizing are modeled from the ground up.

The broader takeaway here is that as AI systems grow more capable, the infrastructure challenges become less about raw model performance and more about intelligent resource orchestration. Caching is one of the most consequential and least discussed levers in that equation. Whether the right answer is extending an existing tool like libCacheSim or building something entirely new remains to be seen, but the fact that practitioners are openly asking for it signals that the industry is ready. The question worth watching is which team — academic, open-source, or inside a major provider — will step forward to define this space the way libCacheSim defined classical cache benchmarking a decade ago.