Developer releases lowvram tool, enabling large models to run on low-VRAM GPUs
Understand this story
Developer Henrique Soares has released lowvram, a command-line tool that intercepts CUDA calls via LD_PRELOAD: it replaces cuMemAlloc with cuMemAllocManaged, allowing applications to swap idle data pages to system memory. According to his introduction, the tool has already run YuE2, which originally required 24GB of VRAM, on an RTX 3060 with 6GB of VRAM. The tool only supports Linux and requires users to be fairly familiar with the CUDA runtime environment. When the model's working set exceeds physical VRAM, data must be transferred between VRAM and system memory over the PCIe bus; because it uses managed memory, if the model continuously requests memory, it may exhaust system memory, since the "available" memory reported by the tool does not decrease as managed memory is used.
AI overview based on source reports · Updated 12 min ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hacker News AILowvram: A CLI tool to run large AI models in low VRAM GPUs
Henrique Soares introduces lowvram, a Linux CLI tool that intercepts CUDA calls via LD_PRELOAD, replacing cuMemAlloc with cuMemAllocManaged so applications can use system RAM when VRAM is insufficient. On a 6 GB RTX 3060, YuE2 under lowvram generated 43.1 tokens per second versus 1.9 for Accelerate and 6.8 for CPU, and Llama-TTS with Qwen3-TTS ran after a standard run failed with an out-of-memory error. The author notes trade-offs: PCIe paging slows computation, system RAM can be exhausted, and the tool requires CUDA knowledge.
Activity for this story
Not enough continuous observations to show a trend yet.