DeepSeek, along with Tsinghua University and Peking University, published a research paper that addresses a major bottleneck in AI inference: the underutilization of GPUs due to inefficient memory access patterns. Their proposed architecture, called Dual Path, can speed up inference by 1.87x and improve online serving throughput by 1.9x.
The Problem: Inefficient GPU Usage in LLM Inference ⏱ 0:32
•Most GPUs have two engines: prefill and decode.•Prefill engine understands the prompt and builds the KV cache.•Decode engine generates the answer token by token using the KV cache.•Prefill engine loads the KV cache from memory; while it does so, the decode engine sits idle.•This underutilization means only a portion of GPU capacity is actively used at a time.•As conversation turns increase (e.g., 30 turns), the context size grows, requiring the previous conversation data for each new turn.•The KV cache stores previous conversation data to avoid recomputation, but the loading mechanism remains a bottleneck.The Current Solution and Its Limitation ⏱ 2:32
•Using KV cache avoids recomputation of previous conversation data.•Prefill engine retrieves the KV cache from memory, then passes it to decode engine.•Decode engine uses the KV cache but does not retrieve it.•This sequential dependency causes decode to idle while prefill loads data.•The diagram shows 100% total work: prefill does most of the work, decode sits idle.The Proposed Dual Path Architecture ⏱ 6:40
•Dual path allows both prefill and decode engines to access the KV storage simultaneously.•This breaks the storage bandwidth bottleneck in agentic LLM inference.•A scheduler prioritizes critical data sharing (e.g., GPU-to-GPU communication) over KV cache access.•The scheduler decides whether prefill or decode accesses the storage based on which is busy.•If prefill is busy, decode handles the access; if decode is busy, prefill handles it.Performance Gains and Implications ⏱ 8:44
•Dual path achieves 1.87x speedup on their in-house inference system.•It also provides 1.9x improvement in online serving throughput.•This is a 2x factor improvement, which is a huge achievement.•With better GPU utilization, fewer GPUs may be needed (e.g., 150 instead of 200).•The key insight: GPU is not a bottleneck; how you manage GPU memory is critical.Key Takeaways
•DeepSeek, Tsinghua University, and Peking University jointly developed a solution to GPU underutilization in LLM inference.•The current architecture forces the decode engine to remain idle while the prefill engine loads KV cache from memory.•Their proposed Dual Path architecture allows both prefill and decode to access the KV cache simultaneously, improving utilization.•The scheduler prioritizes critical GPU-to-GPU data sharing over KV cache access to avoid conflicts.•The method achieves 1.87x speedup and 1.9x improvement in online serving throughput.•Better GPU utilization could reduce the number of GPUs needed for large models.Conclusion
DeepSeek's Dual Path architecture addresses a fundamental memory bottleneck in LLM inference, potentially reducing hardware requirements while improving performance.
