iCreatAI News
Skip to content
vLLM Blog· vLLM Team, Red Hat PyTorch Team, and Meta PyTorch Team·· 1 days agoAI score38

Migrating vLLM to the LibTorch Stable ABI

Migrating vLLM to the LibTorch Stable ABI

AI summary

vLLM describes migrating its C++ backend to the LibTorch Stable ABI so that updating PyTorch no longer requires rebuilding vLLM, and as of v0.33.0 it publishes LibTorch ABI Stable CUDA wheels. The migration involved moving AutogradFunctions and meta functions to Python, building a separate stable library with the TORCH_TARGET_VERSION flag, registering ops via STABLE_TORCH_LIBRARY with TORCH_BOX, and migrating seven LibTorch-dependent dependencies including FlashMLA, DeepGEMM, FlashAttention-2/3, DeepSelect, FlashKDA and QuTLASS. Verification used torch-abi-audit, now in vLLM CI, and the post notes tradeoffs such as the limited stable API subset and small host-side overhead, with CPU and ROCm builds planned next.

Source: vLLM Blog · vllm.ai