vLLM releases ABI-stable CUDA wheel
Understand this story
The vLLM team announced that starting with v0.33.0 it is releasing CUDA wheels based on the LibTorch Stable ABI. After that, when LibTorch is updated, users no longer need to recompile vLLM and can import it simply by replacing the torch wheel. The vLLM team also explained that the LibTorch Stable ABI only solves the binary compatibility problem; vLLM's compatibility with PyTorch is still affected by the Python API, GPU requirements such as the minimum CUDA version, and third-party dependencies; the ABI is also only a limited subset of LibTorch. The vLLM team said the next step is to make all vLLM builds ABI-stable, including CPU and ROCm.
AI overview based on source reports · Updated 55 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- vLLM BlogMigrating vLLM to the LibTorch Stable ABI
vLLM describes migrating its C++ backend to the LibTorch Stable ABI so that updating PyTorch no longer requires rebuilding vLLM, and as of v0.33.0 it publishes LibTorch ABI Stable CUDA wheels. The migration involved moving AutogradFunctions and meta functions to Python, building a separate stable library with the TORCH_TARGET_VERSION flag, registering ops via STABLE_TORCH_LIBRARY with TORCH_BOX, and migrating seven LibTorch-dependent dependencies including FlashMLA, DeepGEMM, FlashAttention-2/3, DeepSelect, FlashKDA and QuTLASS. Verification used torch-abi-audit, now in vLLM CI, and the post notes tradeoffs such as the limited stable API subset and small host-side overhead, with CPU and ROCm builds planned next.
Activity for this story
Not enough continuous observations to show a trend yet.