SGLang 0.5.13 Delivers 25x MoE Inference Speedup via Predictive Routing and Sparse KV Cache
SGLang 0.5.13 introduces two-stage MoE routing prediction and sparse KV cache, achieving a 25x inference speedup on NVIDIA GB300 NVL72. Benchmarks on A100 show 65% throughput gain, 40% latency reduction, and 62% lower routing overhead. This optimization directly attacks the core bottleneck of MoE inference, potentially reshaping AI inference economics.
