Transformers implements MoE expert streaming inference, running a 180B model on a 32GB Mac
Understand this story
Hugging Face has implemented MoE expert streaming loading in Transformers, enabling the 180B-parameter Qwen3.8-Flash-Next to run on a MacBook M5 with 32GB of memory. The author measured short-text decoding at 13.9 tok/s, about 6% slower than Strata's 14.8 tok/s; prefill at 228 tok/s, slightly faster than Strata's 220 tok/s. The implementation is open-sourced as PR #50717; Transformers must be installed from that PR to try it, and the API is still under discussion with maintainers. It applies to MoE models with a standard expert layout; after loading it occupies about 18 GiB, peaking at 25.4 GiB with a 6K prompt.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Hugging Face BlogFeatured180B Parameters, 32GB of RAM: Pushing MoE Inference to the Limit with Transformers
A developer added expert streaming to Transformers via PR #50717, keeping only routed MoE experts in a bounded device cache and reading the rest from SSD, and ran Qwen3.8-Flash-Next (121 GB of FP8 experts, quantized to 3 bits) on a 32 GB Mac M5. After optimizations including Metal kernels and early GPU commits, it reached 13.9 tok/s decode and 228 tok/s prefill, about 6% behind Strata on short decode. The author notes it remains too slow for coding agents and that closing the gap would require per-layer host/GPU sync control PyTorch does not expose.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 6Nov 19 02:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.