One GPU for 100 Simultaneous Requests Without Lag!

The current AI industry competition seems to have fallen into a vicious cycle of "hardware overload," with giants vying for the scale of resource investment and the size of GPU clusters, while few pay attention to the challenges of AI deployment in small and medium-sized scenarios. For enterprises and institutions with small-scale deployments, the real technical bottleneck is not the amount of hardware resources, but:

This pain point directly leads to poor stability of AI services in small clusters and a significantly reduced user experience, becoming a key barrier to the implementation of AI technology in small and medium-sized scenarios.

Addressing this industry gap, UIS Cloud pioneered a breakthrough by developing and launching the P/D virtual partitioning optimisation solution, precisely overcoming the "jitter" problem of mainstream vLLM engines in small-scale clusters and high-concurrency scenarios. The implementation of this solution has resulted in a qualitative leap in the performance of small-scale GPU clusters.

In a single GPU testing environment, the system's response stability under high load has been directly improved by 2–3 times. Previously, when 30 people simultaneously used a single GPU for inference, the latency reached as high as 50ms; however:

This means that even a single GPU server can possess stable inference capabilities comparable to a large GPU cluster. UIS Cloud's innovation not only:

• Breaks the industry's established perception that "hardware scale determines AI service quality"
• Directly lowers the barriers to AI technology implementation in SMEs and niche industry scenarios, making AI accessibility no longer a privilege exclusive to giants, and injecting lightweight and low-cost new possibilities into the intelligent transformation of various industries.

01 — Problems and Current Situation

Challenge 1: Impressive Load Test Data vs. Terrible User Experience

Over the past year, the selling points of large-scale model inference engines have been almost identical:

"Single-machine QPS increased by X times", "GPU utilisation 90%+", "Ultimate throughput"…

These metrics are certainly important, but in a production environment, what users perceive is not "how many tokens each server can process," but rather "how long it takes to see the first response, and whether the subsequent output is smooth."

The industry breaks down LLM inference latency into several key metrics: First Token Latency (TTFT), Average Time Per Output Token (TPOT), and overall end-to-end latency. Among these, TPOT determines the "smoothness" of the user's experience in streaming conversations — each word should be crisp and rhythmic, rather than stuttering like a "typewriter jam."

The reality is that when concurrency increases, although the numbers in many stress test reports look good, the online experience becomes:

• During peak periods, TPOT experienced exponential volatility: the initial token issuance was decent, but subsequent token issuance was sporadic and inefficient.
• In pursuit of maximum throughput, scheduling strategies are very aggressive (such as very large --max-num-batched-tokens). Once the load approaches its limit, long-tail latency skyrockets and SLO (Service Level Objective) is breached.

Ultimately, the business side dared not promise users any definite delays, and could only vaguely say "average" or "okay." This not only brought more uncertainty to the user experience, but also reduced users' trust in the platform to some extent.

Challenge 2: "P/D Separation" Is Not a Silver Bullet for Small-Scale Scenarios

Large model inference is divided into two parts: Prefill and Decode.

• Prefill: Processes the entire Prompt and builds a KV Cache. It involves a large amount of computation and high parallelism, and is a typical compute-bound process.
• Decode: Generated per token, with strong sequential dependency, typically memory-bound.

The characteristics of P and D are highly asymmetrical, so to improve GPU utilisation, inference frameworks now commonly use continuous batching, mixing the P and D of multiple requests in the same batch/in the same iteration. This improves throughput, but also has significant side effects: a long prefill can "occupy the GPU," blocking token generation in the Decode phase, and thus lengthening TPOT and ITL (token interval).

It is now widely recognised in the industry that P/D separation (Prefill-Decode Disaggregation) is a powerful tool to solve inference lag caused by PD interference. This is the standard solution in large-scale public cloud clusters, but for small-scale private deployments, pure PD separation is not a silver bullet and may even be a cost trap.

The resource fragmentation and the operational complexity brought about by P/D separation are often unacceptable. Based on our practical experience, P/D separation has three major drawbacks in small-scale scenarios:

Challenge 3: vLLM Is Powerful, but the Default Strategy May Not Be Suitable for All Scenarios

vLLM itself has undergone extensive engineering optimisations, such as pagination attention mechanism (PagedAttention) and continuous batch processing, which can provide excellent throughput performance in mainstream scenarios.

However, actual business scenarios have two rather "demanding" requirements:

• High load: The number of concurrent requests and the complexity of requests are not low, and the system is often in the medium to high load range. The system can easily be pushed to the "critical point" of throughput and latency.
• Clear TPOT SLO requirements: Not only average TPOT, but also SLO indicators such as average/jitter STD/P99, which need to be observable and commensurable.

In this context, relying solely on the default scheduler strategy is no longer sufficient. We need to further refine the prefill/decode scheduling based on vLLM, truly prioritising TPOT and SLO as front-line metrics, rather than treating them as post-hoc observations.

02 — Design Goal: To Create an Engine That Meets SLO

Yes, we ultimately chose a slightly "counter-trend" path:

Based on the above ideas, we set a very "engineering" goal system for the system transformation: not only to pursue "showy" throughput data, but also to create an inference engine that meets SLO requirements.

03 — UIS Cloud Optimisation Solution: "Minimally Invasive Surgery" for Small Clusters

Compared to the high-throughput strategy of open-source vLLM, PD separation is not flexible enough and has low utilisation in small-scale scenarios. We conducted in-depth analysis and optimisation for this scenario.

We found that the key problem for small-scale clusters is not "the lack of a P/D separation architecture," but "the lack of a flexible scheduling system that can maintain TPOT stability with limited resources." Therefore, we performed a minimally invasive operation on such small clusters, achieving flexible P/D separation through a series of refined modifications:

• Maintain a single instance: Keep resources "on the same set of GPUs," continuing with monolithic instances based on vLLM, and keeping the deployment/operation path/KV transfer complexity as simple as possible.
• Logical PD separation: To avoid interference from the Prefill process to the Decode stage, the execution of P and D is separated from the computing power scheduling, rather than being physically partitioned in the traditional way.
• Dynamic computing power allocation: The scheduler is optimised with a dynamically designed batch strategy. By sensing the current traffic characteristics and the current TPOT performance, it adjusts the allocation of computing power to different P and D executors, maintaining relative stability even when the input/output length distribution is constantly changing.
• Strictly adhere to the allocation ratio: The P/D execution phase is strictly executed according to the computing power allocation ratio to avoid resource contention that could lead to increased Decode latency.

In this way, within the same vLLM instance, by partitioning computing power and separating Prefill/Decode batches, we can maximise the benefits of PD separation at the scheduling layer under the premise of "single machine/small cluster": this avoids interference between Prefill and Decode processes, and also eliminates the need to hard-split the limited GPU machine into two roles.

1. Actual Test Results: TPOT Optimised by 3×

Next, let the data speak for itself:

We used the same hardware environment, the same vLLM base version, and the same testing methods to obtain the results.

a) The Tipping Point of Conflict Between Throughput and SLO

Let's first look at the performance of concurrency (max_concurrency) and average TPOT:

Figure 1: TPOT latency performance under different max_concurrency values

As can be seen, with the increase in concurrency, the latency of the community version of vLLM has reached 80ms when there are 60+ concurrent users. In contrast, the TPOT of UIS Cloud's optimised version of vLLM can always be kept at around 22ms, which is a 3× improvement.

If calculated based on a SLO of 30 tok/user/s, the open-source version of vLLM can support a maximum of around 32 concurrent requests, but UIS Cloud's vLLM can increase concurrent requests without loss of performance.

The above diagram shows max_concurrency, which controls server-side concurrency. When we observe from the perspective of client-side concurrency (request_rate): as shown in the diagram below, even with higher client concurrency, we still have very reliable and stable TPOT performance compared to the open-source vLLM.

Figure 2: TPOT latency performance under different request_rates

Similarly, TPOT with the vertical axis changed to P99 shows a greater improvement compared to the open-source version:

Figure 3: TPOT latency (p99) performance under different max_concurrency values

When we changed the x-axis from "concurrency" to different "input lengths" (ISL), our improvement in performance remained excellent:

Figure 4: TPOT latency (p99) performance under different input lengths

b) The Silky Smoothness Felt by the User

Let's take a look at the stability performance of TPOT (the following figure shows the standard deviation (jitter level): std_tpot):

Figure 5: TPOT standard deviation under different max_concurrency values

As can be seen, compared to the open-source version of vLLM, UIS Cloud's TPOT performance is extremely stable, with a standard deviation of less than 4ms, while the open-source version's standard deviation is around 15ms. In comparison, the TPOT jitter is three times greater. This indicates that using UIS Cloud's vLLM results in exceptionally stable output.

It means that the rhythm of each word's pronunciation is uniform, which is the core of the user's perception.

c) Cost: How Much Throughput Was "Sacrificed"?

Of course, as mentioned earlier, since computing power is limited, to guarantee SLO, some throughput needs to be sacrificed:

Figure 6: Throughput performance under different request_rate settings

As shown in the graph, under low load conditions, the throughput difference compared to the open-source vLLM is approximately 8%. Under high concurrency conditions, the throughput difference does not exceed 25%, but in return, it results in a 2–3 times increase in TPOT and SLO guarantee.

While the throughput of the open-source vLLM increases with concurrency, it comes at the cost of TPOT (user experience). (For example, high throughput achieved under conditions of very high TPOT is considered "ineffective throughput," and users are likely to churn due to lag.)

Figure 7: Throughput performance under different TPOTs

Conversely, UIS Cloud's modified vLLM allows for stable TPOT while maintaining throughput, ensuring a superior user experience. Within the effective range that meets SLO (e.g., TPOT < 30ms), our actual load capacity is significantly stronger.

04 — Summary

Say goodbye to "settling" and embrace control. The dilemma of small-scale, large-model inference is being broken by new methodologies.

Our vLLM TPOT optimisation solution is just the beginning, but it clearly points in a direction:

Through out-of-the-box optimisation, we achieved a leap in TPOT, long-tail convergence, and SLO observability, proving that "large-scale" experience is not the exclusive domain of tech giants.

This is not just a performance verification, but a ticket to the future. It invites every developer to rethink: how to use more sophisticated engineering wisdom to create unlimited possibilities with limited resources. The road has been paved; it awaits your exploration.