iCreatAI News
Skip to content
vLLM Blog· Felix Li, Lingpeng Jin, Hattie Wu, Carlus Huang, Fangzhou Ai, Zaidy Ali, Amanzhol Salykov, Hyukjoon Lee, Ziming Huang, Jun Chen, Yifeng Huang, Bin Ding, Xiaobing Zhang, Yi Gan, Vinayak Gokhale, Hongxia Yang, Doug Lehr, Richard Li, Andy Luo, Peng Sun, and Emad Barsoum·· 3 days agoAI score58

Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM

Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM

AI summary

This post describes how MonoKernels written in FlyDSL, AMD's Python DSL and MLIR compiler for ROCm kernels, reduce decode latency in vLLM for interactive LLM serving on AMD Instinct GPUs by combining dependent operator stages into persistent single-kernel launches with thread-block reuse, weight prefetch, and tagged-mailbox handoffs. Mono decode is opt-in via --monokernel and merged in vLLM main for DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3, and Muse Spark on MI355X (plus MI325X for DeepSeek-V4.1-Flash), reporting concurrency-1 speedups of 1.20× to 2.26× with accuracy differences within noise for the listed configurations. The post also notes limits: gains are largest at low concurrency, step widths are capped (e.g., 48 rows for DeepSeek-V4.1-Flash on MI355X), and MI300X/MI325X keep separate weight copies.

Source: vLLM Blog · vllm.ai