iCreatAI News
Skip to content
Hugging Face Blog·· 3 days agoFeaturedAI score60

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers

AI summary

A developer added expert streaming to Transformers via PR #50717, keeping only routed MoE experts in a bounded device cache and reading the rest from SSD, and ran Qwen3.8-Flash-Next (121 GB of FP8 experts, quantized to 3 bits) on a 32 GB Mac M5. After optimizations including Metal kernels and early GPU commits, it reached 13.9 tok/s decode and 228 tok/s prefill, about 6% behind Strata on short decode. The author notes it remains too slow for coding agents and that closing the gap would require per-layer host/GPU sync control PyTorch does not expose.

Why it matters

Useful for engineers weighing MoE offloading strategies across llama.cpp, vLLM and Transformers, with concrete benchmarks and an honest account of the remaining bottleneck.

Source: Hugging Face Blog · huggingface.co