Llm Inference Lmcache

The rapid growth of AI Agent technology has created an urgent demand for handling extremely long contexts and large volumes of information, pushing LLM inference to face severe bottlenecks in both performance and compute. Current mainstream KV-Cache approaches can speed up inference by caching intermediate data in GPU memory, but the limited and expensive GPU memory they consume has become the key bottleneck constraining efficient inference.

To address this challenge, Nvidia, UIS Cloud, and partner companies, leveraging advanced LMCache high-efficiency caching technology from the open-source community, built a hardware-software co-optimisation architecture to achieve high-efficiency distributed storage and scheduling of KV Cache. This significantly alleviates GPU memory pressure while effectively improving model generation speed. In collaboration with vendors including Chaoji Zhiyan and Lenovo Tianzhi, tens of millions of RMB in equipment and resources were invested for optimisation testing.

~24x

In typical medium-length context scenarios (approx. 10K tokens), TTFT (Time to First Token) is greatly reduced, and throughput achieves a nearly 24x improvement. This lays a solid foundation for stable production deployment of KV Cache tiered-storage solutions and scalable LLM inference in long-context scenarios.

01 — KV-Cache and Its Application Challenges

KV-Cache (Key-Value Cache) is one of the core data structures in the attention mechanism used during LLM inference. For example, if you give the model the input: "I want to eat fruit, recommend a few kinds" (assume 7 tokens), the process has two stages:

Prefill Stage: The model processes all 7 tokens at once — computing the Key and Value for each token, then storing them all in KV Cache. This step "fully digests" your initial context, equivalent to "taking down key information notes".
Decode Stage: The model generates the first reply token, e.g. "Apple" (1 token), computing and storing its K and V (now 8 groups). To generate "Orange" next: only "Orange"'s K and V need computing — the existing 8 KV groups are fetched directly from Cache. This continues until the full reply is generated.

Throughout this process, KV-Cache acts like a "real-time updating notepad": the initial input is memorised all at once, and thereafter each new token only adds new KV entries while previous information is reused — saving a great deal of redundant computation.

The problem: the longer your input and output, the more KV pairs accumulate, filling GPU VRAM. LMCache addresses this by offloading part of the cache data from GPU to CPU memory or remote storage (e.g. NVMe-oF, network storage).

To validate these capabilities in production, Nvidia, UIS Cloud, Chaoji Zhiyan, and Lenovo Tianzhi conducted a joint hardware-software co-optimisation benchmarking exercise. UIS Cloud was primarily responsible for test design, model deployment, LMCache cache strategy configuration, load-test execution, and performance data analysis. Nvidia provided GPU, switches and hardware support; Chaoji Zhiyan supplied high-performance GPU servers and test environments; Lenovo Tianzhi supplied high-performance storage servers.

02 — Test Setup

This benchmark simulates enterprise multi-turn dialogue scenarios.

Test Environment

Chaoji CQ7688-L × 1: Equipped with NVIDIA H20 GPU; 8U 8-GPU NVLink; supports 4.0 Tbps network bandwidth; includes 8 east-west 400 Gbps Bluefield-3 SuperNIC plus 2 south-north 400 Gbps BlueField-3 DPU.
ROCE Switch × 6: NVIDIA 5th-generation Spectrum Ethernet switches; both compute network and storage network use SN5600.

Storage Servers

Lenovo Tianzhi WR5220 G3 high-performance storage server, mounted locally via RDMA-based NFS protocol, maximum read bandwidth up to 400 Gbps.
RDMA-based NVMe-oF protocol mounted to local network storage, maximum read bandwidth up to 800 Gbps.

Figure 1: Three-Tier KV-Cache Architecture — L1: GPU VRAM | L2: RAM | L3: Lenovo WR5220 G3 (400Gbps) & NVMe-OF (800Gbps) via BlueField NIC & Spectrum SN5600

Software Platform: UIS Cloud's d.run compute scheduling platform.

Model Deployment

Model DeepSeek-R1-0528 deployed using image lmcache/vllm-openai:v0.3.3, vllm version v0.10.0.x, CUDA 12.8. TP8 deployment; vllm prefix caching disabled to reduce test interference. Single-machine combined P/D configuration, launched via Kubernetes.

// Key Deployment Parameters

image: lmcache/vllm-openai:v0.3.3
--tensor-parallel-size "8"
--no-enable-prefix-caching
--kv-transfer-config '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}'

LMCache Configuration

LMCache provides two IO modes for storage read/write:

LocalDisk Mode ("Non-GDS Mode"): A general, highly compatible storage IO method — any file system path accessible to the OS, including remotely-mounted local storage. Data path: GPU → CPU → Storage.
GDS Mode: Uses NVIDIA's GPUDirect Storage (GDS) technology to transfer data directly from compatible storage to GPU VRAM, bypassing the CPU — achieving lower latency and higher throughput.

// CPU-Offload mode (RAM only as L2 cache)

LMCACHE_USE_EXPERIMENTAL=True
LMCACHE_CHUNK_SIZE=256
LMCACHE_MAX_LOCAL_CPU_SIZE=100

// GDS (GPU Direct Storage) mode

LMCACHE_CUFILE_BUFFER_SIZE="8192"
LMCACHE_LOCAL_CPU=False
LMCACHE_GDS_PATH="/mnt/cache/deepseek-r1-gds/"

03 — Test Steps and Results

Test Objectives

Validate performance gains from enabling KV Cache
Test performance changes from different storage IO modes
Test performance gains from different network bandwidths under the same storage IO mode

Key Metrics

TTFT (Time to First Token)

The primary metric. By reusing KV Cache, Prefill recalculation is eliminated, dramatically reducing time to first token.

Total Token Throughput

As TTFT decreases, overall system throughput increases proportionally.

Context Lengths Tested

Context lengths sampled: 100, 1K, 10K, 50K, 100K tokens. Reference: multi-turn dialogue averages 500–2K tokens (early), 5K–10K tokens (later); AI-assisted coding averages 4K–16K tokens; RAG typically 8K–32K tokens.

For each length and backend, the same input was tested 10 rounds; rounds 2–9 results were averaged to exclude cold-start bias.

Results — Comparison of Different Storage Backends

LMCache supports different storage backends. Benchmarking compared four scenarios:

Baseline (Red): No KV Cache reuse
CPU (Green): CPU RAM as KV-Cache Offload, 100 GB/rank
Non-GDS (Dark Blue): 400 Gbps network storage using Non-GDS mode
GDS (Cyan): 400 Gbps network storage using GDS mode

Figure 2: Mean TTFT vs Input Length — comparison across different storage backends (lower is better)

Figure 3: Token Throughput vs Input Length

Figure 3: Total Token Throughput vs Input Length — comparison across different storage backends (higher is better)

Key Finding

Compared to no KV Cache reuse, KV-Cache Offload delivers a maximum throughput improvement of nearly 24x — occurring at 10K token context length using CPU offloading. As context exceeds 100K tokens, CPU RAM becomes exhausted and its advantage disappears. This is where large-capacity network storage excels — in ultra-long contexts, high-speed Nvidia Spectrum-backed network storage achieves throughput gains of nearly 10x.

Test results also show that GDS storage IO mode consistently outperforms Non-GDS mode, with throughput improvements of 32.5%–96.6%. This demonstrates that in high-bandwidth, low-latency network storage systems, LMCache better exploits the advantages of its heterogeneous cache layers.

Figure 4: Throughput improvement vs. no KV Cache across different storage backends and input lengths

Results — Impact of Network Bandwidth

Tests compared performance under different network bandwidths using GDS mode with NVMe-oF storage: 100 Gbps vs. 400 Gbps vs. 800 Gbps configurations. Results demonstrate that higher bandwidth storage directly translates to improved KV Cache retrieval throughput, especially pronounced at longer context lengths (50K–100K tokens) where cache data volume is largest.

Figure 5: Mean TTFT vs Input Length — bandwidth comparison

Figure 5: Mean TTFT vs Input Length — non-gds vs gds at 400g and 800g (lower is better)

Figure 6: Token Throughput — bandwidth comparison

Figure 6: Total Token Throughput vs Input Length — non-gds vs gds at 400g and 800g (higher is better)

Figure 7: Throughput comparison — disk/gds 400g vs 800g across input lengths

Architecture Summary: Three-Tier KV-Cache

GPU VRAM

Fastest tier — active KV pairs for current inference. Limited capacity (~80–140 GB per H20 GPU).

CPU RAM

High-capacity overflow tier — up to 100 GB/rank. Effective for medium-context workloads (up to ~50K tokens).

Network Storage (NVMe-oF / NFS via RDMA)

Lenovo WR5220 G3 at 400 Gbps (NFS) or 800 Gbps (NVMe-oF). Critical for ultra-long contexts exceeding CPU RAM capacity.

Conclusion

This collaborative benchmarking by Nvidia, UIS Cloud, Chaoji Zhiyan, and Lenovo Tianzhi demonstrates that hardware-software co-optimisation with LMCache delivers transformative gains in LLM inference efficiency. The three-tier KV-Cache architecture effectively removes GPU VRAM as the primary bottleneck, enabling scalable long-context LLM inference without proportional hardware cost increases.

Key takeaways for production deployments:

CPU offloading delivers the highest gains (~24x) for typical enterprise workloads (5K–15K tokens)
High-speed RDMA network storage enables extreme long-context scenarios (50K–100K+ tokens)
GDS mode consistently outperforms Non-GDS by 32.5–96.6% — use GDS where possible
UIS Cloud's d.run platform provides the scheduling foundation for production-grade KV Cache deployments

LLM Inference Performance Surges Nearly 24x