Large-Scale Model Inference Has Achieved Rocket Speed:

Introduction

Why does AI reply to messages as slowly as a typewriter? The answer is simple: insufficient video memory. Video memory is not only expensive but also extremely scarce, making it the "number one bottleneck" for AI reasoning. Especially when processing long articles or long conversations, the temporary data generated by the AI's thinking in the KVCache (key-value cache) will quickly fill up the video memory. This directly reduces the number of users that can be served simultaneously, and in severe cases, it will freeze completely and cannot even produce a reply.

To address this pain point, UIS Cloud, in collaboration with core partners such as NVIDIA, SuperEngine Intelligence, and SuperCloud, has provided a systematic solution with immediate and effective results:

Combining these two technologies simultaneously overcomes two long-standing challenges: it fundamentally solves the problem of insufficient GPU memory capacity and addresses the issue of uneven computing resource allocation. Actual testing shows that in an AI inference scenario with two GPU servers and hundreds of concurrent users, model generation speed is increased by up to 50 times, first token response time is reduced by up to 75%, and throughput is increased by up to 4.8 times. This result provides a clear and feasible path for the development of AI infrastructure for inference. In the future, enterprises can expect to deploy and expand AI services at a lower cost, allowing larger, more intelligent models to be applied more smoothly across various industries.

01 — From Stand-Alone to Multi-Machine Collaboration

Previously, UIS Cloud had independently verified two key technologies — "KVCache Offloading" and "PD Separation" — to address performance bottlenecks in large model inference, both of which achieved significant results.

Regarding KVCache Offloading:

Previously, UIS Cloud, together with partners such as NVIDIA and SuperQ Intelligence, offloaded KVCache to the CPU, which significantly shortened the first token response time and increased the overall throughput by nearly 24 times in medium-length contexts (about 10K tokens).

With increased concurrency and longer context lengths, CPU and memory became limited. UIS Cloud opted to offload KVCache to an external storage server and introduced GDS (GPUDirect Storage) technology to accelerate data access. The same tests were conducted based on a combined GDS + cloud storage + KVCache offloading solution. Compared to the vLLM baseline solution, for long input text (50K tokens ~ 100K tokens), the first token latency (TTFT) was reduced by up to 97%, and throughput increased by up to 39 times.

Regarding PD Separation:

UIS Cloud conducted performance tests on the P/D virtual partitioning mechanism to address the response jitter issue of the vLLM engine in high-concurrency, small-scale GPU clusters. In high-concurrency tests on a single GPU, system stability was improved by 2–3 times, and the inference latency for 128 concurrent users was reduced from 50ms before optimisation to less than 22ms. This achieved a smooth inference experience close to that of large-scale clusters with extremely low hardware requirements, laying a technical foundation for the lightweight popularisation of AI inference.

The above verifications were all conducted on a single GPU server. However, in real-world production scenarios, a single GPU server often cannot meet the computing power demands. Based on the significant advantages demonstrated by the two solutions, UIS Cloud planned to further test the combined effect of the two solutions on multiple GPU servers.

When performing inference on multiple GPU servers, the bottleneck of CPU memory capacity becomes more apparent. Therefore, UIS Cloud prioritised external high-performance storage for the KVCache and further improved storage integration technology. More importantly, by introducing and refining the PD separation architecture, UIS Cloud not only optimised throughput but also raised the response speed of the model service to a new standard.

02 — The Combined Effect of the Solutions is Greater Than the Sum of its Parts (1+1>2)

UIS Cloud, in collaboration with NVIDIA, SuperQ, SuperCloud, and Nadode, deeply integrated PD separation with KVCache Offloading to perform performance verification of contexts of varying lengths (100–50K Tokens) in a two-GPU server inference environment. This involved experimental design, environment setup, execution testing, and analysis of experimental results in different scenarios.

GDS and Weka GDS

When unloading KVCache, the common practice in the industry is to unload KVCache to CPU memory. However, testing found that unloading KVCache to remote storage outperforms unloading it to CPU memory. The solution of using GDS + remote storage for KVCache performs better.

The key advantages of using GDS + remote storage for KVCache are:

• Data is directly DMA-transferred from remote storage into GPU memory, bypassing the CPU/kernel path.
• Significantly increased bandwidth and reduced latency.
• Reduced CPU usage and support for efficient shared cache.

Compared to traditional methods, this approach offers substantial improvements in latency, bandwidth, CPU utilisation, and multi-node sharing capabilities.

Especially after introducing the latest Weka GDS technology, this solution achieves a significant performance improvement. This improvement is mainly attributed to Weka GDS's enhancements in network optimisation and data storage processing. Through efficient data distribution and caching strategies, it better adapts to large-scale concurrent workloads, thus handling more requests under the same network conditions. Compared to the standard GDS protocol, it makes better use of network bandwidth and storage resources, reducing bottlenecks in data transmission and processing.

Therefore, in this test, UIS Cloud focused on the KVCache offloading solution based on GDS and Weka technologies.

Test Environment and Configuration

Hardware Environment:

Software Environment:

Test Configuration

UIS Cloud set up full-sequence input scenarios ranging from short to long (100 to 50,000 tokens) and used fixed concurrency and rate for stress testing to ensure data comparability.

Basic Input Parameters:

The test is divided into three modes: Prefill-only, Decode-only, and PD (complete process).

Test Results

This test compared five architecture schemes, covering traditional coupled architecture, KV Cache Offload CPU, and three different PD separation combinations, and measured them from three core dimensions: Throughput, First Token Latency (TTFT), and Token Generation Time (TPOT).

Architecture Scheme Definitions:

Figure 1: Performance comparison of First Token Delay (TTFT) under different input sequence lengths

Figure 2: Performance comparison of single token generation time (TPOT) under different input sequence lengths

Figure 3: Performance comparison of throughput metrics under different input sequence lengths

Core Data Conclusions

Compared to the CPU-offload approach, all three PD separation schemes show significant improvements in inference performance:

03 — Future Thinking: The Choice Between Separation and Collaboration

The performance challenges of large-scale, long-context inference have evolved into an extremely complex end-to-end system-level engineering project. In this test, UIS Cloud fully presented the optimisation path for large-model inference centred on "PD separation" and "KV Cache Offload" and its enormous potential. At the same time, the test data also revealed deeper system insights: the full release of the performance of the PD separation architecture is highly dependent on whether the supply capacity of Prefill and the consumption capacity of Decode can form an efficient pipeline.

The diminishing marginal performance observed in scenarios with extremely long contexts (such as 50K tokens) stems from the fact that the Prefill phase must simultaneously handle heavy computation and data transmission tasks, causing its supply rate to lag behind the high throughput demands of the Decode side, thus creating a supply bottleneck. Successful system design must strike a precise balance between a unified PD (resource aggregation) and a separate PD (physical isolation) architecture, based on the differentiated business requirements for "total throughput" and "interaction latency."

This points the way for next-generation optimisation: future breakthroughs will not only lie in "separation" but also in "synergy" — through more granular KVCache scheduling, finer-grained, streaming producer-consumer overlap, and continuous optimisation of the storage backend, further improving the overall throughput of the PD separation architecture in ultra-long context inference.