Cloudian benchmarks of S3-native object storage with RDMA acceleration demonstrate up to 20x lower TTFT versus recompute, and a 31.77% mean latency advantage versus TCP across tested ISL sizes at P95.
As large language models grow in capability, they’re also growing in cost. And not just the cost of GPUs. Every time a user returns to a conversation, uploads a document for analysis, or triggers a multi-turn agentic workflow, the model has to rebuild its internal state from scratch. This reconstruction, known as prefill, is the single most expensive computation in LLM inference. It burns GPU cycles, delays responses, and limits how many users a system can serve.
There’s a better way. By offloading the model’s key-value cache to high-performance external storage and reloading it on demand for cache-hit requests, organizations can avoid redundant prefill computation and reduce time to first token. The challenge has always been finding a storage tier fast enough to make this practical. Our testing with NVIDIA Dynamo shows that Cloudian HyperStore, using RDMA-accelerated S3 object storage, is more than fast enough.
What Is the KV Cache and Why Does It Matter?
When a large language model processes a prompt, it computes a set of key-value pairs for every token at every layer of the model. These KV pairs represent the model’s “memory” of the conversation — its understanding of the context. For a large model like Llama 3.1-405B processing a full 128K-token context window, this cache grows to approximately 61 GB of data.
Today, most inference systems treat the KV Cache as ephemeral. When a session ends or the GPU needs memory for other users, the cache is discarded. The next time that user returns, even with an identical prompt or a continuation of the same conversation, the model has to recompute everything from the beginning. This full prefill pass is computationally intensive and time-consuming, often taking well over a minute for long-context requests.
KV Cache offloading changes this equation entirely. Instead of discarding the cache, it’s saved to external storage. When the same context is needed again, the pre-computed cache is loaded directly back into GPU memory, bypassing the expensive prefill computation altogether. The result is dramatically faster Time to First Token (TTFT) and significantly higher GPU utilization, since those compute cycles can now serve other requests instead of recalculating work that’s already been done.
Cloudian’s Test: Measuring the Real-World Impact
To quantify the benefit of KV Cache offloading with object storage, we ran a benchmark using the NVIDIA Dynamo inference framework with its KV Block Manager (KVBM) and Cloudian’s RDMA-accelerated plugin for NVIDIA NIXL.
The software stack consisted of vLLM as the inference engine, Dynamo KVBM for cache management, and Cloudian HyperStore as the external storage tier, connected via S3 RDMA for direct GPU-to-storage data transfer. The client system was equipped with 8x NVIDIA RTX PRO 6000 GPUs and 2x 200 Gbit/s network links bonded to provide 400 Gbit/s of aggregate bandwidth.
The test isolates KV Cache offload performance by sending the same large-context request to the model twice. On the first pass, the model performs full prefill computation, generating the KV Cache from scratch across all layers and saving it to Cloudian object storage. On the second pass, the request is identical, producing a 100% cache hit. The entire KV Cache is read back from storage rather than recomputed. The difference in TTFT between the two passes directly measures how effectively the storage tier can serve cached data back to the GPUs.[1]
Results
The data reveals two distinct stories. First, both S3 retrieval methods deliver dramatically lower TTFT than full recompute across every context length tested. At 120K tokens, recompute latency reaches 158.86 seconds at P95, while RDMA delivers the same result in just 7.59 seconds. That’s a 20x reduction in TTFT for long-context workloads.
Second, the RDMA advantage over TCP grows significantly with context length. At 1K tokens the two protocols perform nearly identically, but by 60K tokens RDMA is already 41% faster, reaching a 40.56% advantage at 120K tokens and a mean improvement of 31.77% across all ISL sizes tested.
For the large-context workloads that define production AI inference today, RDMA’s direct GPU-to-storage path is not a marginal upgrade. It is a significant architectural advantage.
Performance Across Context Lengths: The Cold Cache Scenario
The benchmark examines performance across the full spectrum of context lengths, from 1K to 120K tokens, under the cold cache miss condition, where the KV Cache is not present in local CPU memory and must either be recomputed from scratch or retrieved from remote S3 storage. Three conditions are compared at the P95 percentile: full recomputation with no cache available; retrieval from Cloudian S3 storage via RDMA; and retrieval via standard TCP/IP.
Three conditions are compared at the P95 percentile for Time to First Token (TTFT): full recomputation with no cache available; retrieval from Cloudian S3 storage via RDMA; and retrieval via standard TCP/IP. The results are shown in the chart below.
The data reveals a critical divergence. Recompute latency scales linearly with context length: at 1K tokens TTFT is 3.88 seconds, but by 100K tokens it reaches 129.59 seconds at P95 — and climbs further to 158.86 seconds at 120K. By comparison, the latency curves for both storage retrieval methods are essentially flat compared to recompute. The cost of cache retrieval does not scale with context length the way recomputation does. S3 over RDMA stays below 8 seconds across the entire range tested, while S3 over TCP holds under 13 seconds even at the longest context lengths.
If we zoom in to focus specifically on the TCP vs RDMA data, we see another important element of the story. The comparison between RDMA and TCP shows that removing the CPU from the data path matters at scale. At 1K tokens the two protocols perform nearly identically, a difference of less than 50ms.
But the gap grows consistently with context length. At 60K tokens RDMA is 41% faster than TCP; at 80K tokens, 44%; at 120K tokens, 40.56%, with a mean advantage of 31.77% across all ISL sizes tested. RDMA’s direct GPU-to-storage path delivers a measurable and growing latency advantage as data volumes increase.
What This Speedup Actually Means
Raw benchmark numbers are useful, but the real impact of KV Cache offloading shows up in how inference systems operate at scale.
Multi-turn conversations become seamless. Enterprise AI applications increasingly involve extended interactions, such as a legal analyst working through a lengthy contract, a developer debugging across multiple exchanges, or an AI agent executing a complex multi-step workflow. Without cache offloading, every turn in the conversation forces the model to rebuild context from the beginning, adding latency that compounds with each exchange. With Cloudian serving the cached state in seconds, returning to a prior conversation can feel much faster across long-context workloads.
GPU utilization improves dramatically. Prefill computation monopolizes GPU resources. Every second a GPU spends recalculating a KV Cache is a second it can’t spend serving other users. By eliminating redundant prefill operations, cache offloading frees GPU cycles for productive inference work. At a 120K-token context, the 158 seconds of GPU compute time saved per cache hit can instead serve dozens of additional requests — a direct improvement in throughput and cost efficiency.
Long-context applications become practical. Models with 128K+ token context windows unlock powerful capabilities: entire codebases for analysis, full legal documents for review, comprehensive financial reports for summarization. But the compute cost of prefilling these massive contexts has been a barrier to deploying them at scale. When the cached context can be reloaded from storage in seconds rather than recomputed over minutes, organizations can offer long-context capabilities to large user populations without proportional increases in GPU infrastructure.
RAG and document analysis workflows accelerate. In retrieval-augmented generation workflows, the same base document context is often reused across many queries from different users. With KV Cache offloading, the first user to process a document pays the prefill cost once. Every subsequent query against that same document context can load the pre-computed cache from Cloudian in seconds, turning what was a per-query compute expense into a shared, amortized storage cost.
Why Object Storage Is the Right Tier for KV Cache
The choice of storage platform for KV Cache offloading has implications well beyond benchmark performance. Object storage brings architectural advantages that align naturally with the demands of large-scale inference infrastructure.
Exabyte scalability for an exabyte problem. KV Cache data grows with users, context length, and model size. In this test scenario, a single 128K-token session on a large model produces roughly 61 GB of cache. Multiply by thousands of concurrent users, factor in session history for multi-turn reuse, and the storage requirement reaches petabyte scale quickly. Object storage’s flat namespace and horizontal scaling architecture handles this growth without the management complexity or performance cliffs of file-based systems.
S3 is the universal data language. The S3 API is the de facto standard for unstructured data access across on-premises, cloud, and hybrid environments. By using S3-compatible storage for KV Cache, organizations leverage the same storage platform — and the same operational tooling, policies, and expertise — they already use for training data, model artifacts, and application data. There’s no need for specialized file systems or additional protocol infrastructure.
RDMA removes the CPU from the data path. Cloudian’s implementation of NVIDIA’s RDMA for S3-compatible storage establishes a direct data highway between storage and GPU memory, bypassing the host CPU entirely. Traditional TCP-based transfers require the CPU to process every packet, copy data through system memory, and manage protocol overhead. RDMA eliminates all of this, delivering higher throughput and lower latency. Those freed CPU resources are available for other inference pipeline tasks.
Enterprise data governance comes built in. KV Cache isn’t just computational state — it can contain sensitive conversational context, proprietary document content, and user-specific information. Object storage provides the encryption, access controls, immutability policies, and audit capabilities that enterprise security and compliance teams require. For organizations building sovereign AI infrastructure or operating under data residency regulations, this isn’t optional, it’s foundational.
The Path Forward: KV Cache as Infrastructure
KV Cache offloading is emerging as an important production consideration for long-context and multi-turn inference workloads. NVIDIA’s investment in NVIDIA Dynamo and NVIDIA NIXL, the development of RDMA for S3-compatible storage, and the broader industry momentum around disaggregated inference all point in the same direction: the storage tier is becoming a first-class component of the inference architecture.
Our results demonstrate that this tested configuration of Cloudian HyperStore S3 object storage with RDMA acceleration delivers the throughput needed to serve KV Cache at the speed GPUs demand, while providing the scalability, standardization, and data governance that enterprise deployments require.
As context windows grow, multi-turn interactions deepen, and agentic AI workflows become the norm, the ability to persist and recall model state efficiently will separate performant inference platforms from those that waste GPU cycles recalculating what they’ve already computed. With RDMA delivering up to 20x lower TTFT versus recompute and a 32% latency advantage over TCP at long contexts, the benchmark results point to a strong performance case: offloading KV Cache to Cloudian object storage with RDMA acceleration can improve performance for cache-hit, long-context workloads.
Test Configuration
| Parameter | Detail |
| Storage | Cloudian HyperStore (S3 object, RDMA-accelerated) |
| GPUs | 8x NVIDIA RTX Pro 6000 |
| Network | 2x 200 Gbit/s bonded (400 Gbit/s aggregate) |
| Software stack | vLLM + NVIDIA Dynamo KVBM + Cloudian RDMA-accelerated NIXL plugin |
| Object size | 8 MiB |
| Benchmark | NVIDIA Dynamo KV Cache offload benchmark |
Learn more about Cloudian solutions for AI here.
Schedule a call with an AI expert here.
[1] Test software: NVIDIA Dynamo inference platform ; AWS S3 C++ SDK that transfers data over RDMA; modified Milvus vectorDB (uses AWS SDK for S3 data transfers), integrated with cuVS to offload index and search functions to the host GPUs.
Dataset or prompt shape: English version of the miracl corpus to Milvus (https://huggingface.co/datasets/miracl/miracl-corpus – a 265GB dump of Wikipedia containing 32,893,221 passages across 5,758,285 articles).
Cache-hit assumption: 100% during runs with KVcache enabled.
Test site: NVIDIA lab
