Inkling 2 Small on vLLM: Day-0 Support
Inkling 2 Small on vLLM: Day-0 Support
vLLM announces day-0 support for Thinking Machines' Inkling 2 Small, a multimodal text-and-vision MoE model with 594B total and 17B active parameters, up to 1M tokens of context, and a native DSpark speculator. The post details the architecture (48 layers, 384 experts with top-8 routing, block attention residuals, gated attention, a 23-layer ViT image encoder, and mixed NVFP4/MXFP8 expert precision) and vLLM's optimizations, including kernel fusion with PDL, fused MoE activation quantization, sequence parallelism for prefill, and a Gluon relative attention kernel, reaching up to 630 TPS/user at TP4 on GB200. It also covers LoRA and RL (via SkyRL) support, a serve command, agentic-workload benchmarks, and accuracy checks on NIAH up to 1M tokens, MMMU-Pro and HLE.
Useful for engineers evaluating or deploying a large multimodal MoE model, as it pairs concrete architecture and quantization details with vLLM-specific kernels, benchmark numbers and a ready serve command.
Source: vLLM Blog · vllm.ai