Disaggregated Prefill and Decode
Separating prefill and decode across devices speeds up LLM inference and improves performance
TL;DR
- LLM inference is split into prefill (input tokens) and decode (token generation) stages.
- Running prefill and decode on separate devices (disaggregation) maximizes throughput and minimizes latency.
- This separation involves prefiller nodes populating KV caches and transferring them to decoder nodes.
- A KV messenger orchestrates KV cache transfers, utilizing RDMA (EFA, ConnectX) and libfabric for low-latency communication.
- Sharded KV cache transfers are managed by a transfer engine that coordinates multiple devices, adapting to different sharding schemes.
- Speculative decoding requires minor adjustments, with prefill not including the last token to simplify implementation.
- Disaggregated deployments have been tested with models like DeepSeek-R1 and Qwen3-Coder, showing significant TPS improvements compared to mixed prefill-decode.
- The DeepSeek-R1 deployment showed over 90 TPS with disaggregation versus less than 50 TPS when mixed.
- Qwen3-Coder utilized GQA and sharded KV caches with the HND layout for efficient parallel transfers.