Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM
Keeping Agents Fast: High-Interactivity Decode with FlyDSL MonoKernels in vLLM
This post describes how MonoKernels written in FlyDSL, AMD's Python DSL and MLIR compiler for ROCm kernels, reduce decode latency in vLLM for interactive LLM serving on AMD Instinct GPUs by combining dependent operator stages into persistent single-kernel launches with thread-block reuse, weight prefetch, and tagged-mailbox handoffs. Mono decode is opt-in via --monokernel and merged in vLLM main for DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3, and Muse Spark on MI355X (plus MI325X for DeepSeek-V4.1-Flash), reporting concurrency-1 speedups of 1.20× to 2.26× with accuracy differences within noise for the listed configurations. The post also notes limits: gains are largest at low concurrency, step widths are capped (e.g., 48 rows for DeepSeek-V4.1-Flash on MI355X), and MI300X/MI325X keep separate weight copies.
Source: vLLM Blog · vllm.ai