Language & LLMs
vLLM improved serving throughput for popular large language models by two to four times
In a 2023 paper, researchers introduced PagedAttention, a memory-management method for large language model serving, and a serving system called vLLM. The paper reports that vLLM improved throughput by 2 to 4 times at the same latency compared with FasterTransformer and Orca. Throughput is the number of requests handled in a given time, and latency is the response delay.
Why does the key-value cache become a bottleneck?
Each request needs a large temporary store called a key-value cache. That store grows and shrinks while the model answers. When the cache is managed poorly, fragmentation and duplicate copies waste memory. That waste limits how many requests can be batched at one time, which in turn limits throughput.
How does PagedAttention manage memory?
PagedAttention borrows the idea of paging from operating systems. In paging, memory is divided into blocks so programs can use it more flexibly. The paper says vLLM reaches near-zero waste in cache memory and allows the cache to be shared within and across requests, which further reduces memory use.
What did the evaluation show?
The authors compared vLLM with FasterTransformer and Orca, two existing serving systems. They report that vLLM improved throughput for popular large language models by 2 to 4 times while keeping the same level of latency. The improvement was larger for longer sequences, larger models, and more complex decoding algorithms, which are the methods a model uses to choose its next words.
Questions about the paper
What is paging?
Paging is an operating-system technique that divides memory into blocks. A program can then use those blocks flexibly. PagedAttention applies that idea to the key-value cache used in language model serving.
Which systems were compared?
The paper compares vLLM with FasterTransformer and Orca. The evaluation covered popular large language models, though the abstract does not name individual model versions.
Is the code available?
The abstract states that vLLM's source code is publicly available.
Limits and what's next
The abstract does not state limitations.
Source: Woosuk Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” 2023.