NVIDIA demonstrates three optimization methods for MoE inference on DGX Spark
Understand this story
An NVIDIA technical blog demonstrated three optimization methods for accelerating Mixture-of-Experts (MoE) model inference on a single DGX Spark, and said they increased the throughput of Inclusion AI's Ling-3.0-flash by 42.5%. The model has 124B total parameters, and the evaluation was conducted with a batch size of 1. Earlier performance analysis showed that the decoding phase is limited by memory bandwidth, with reading model weights dominating execution time, and two operations—MoE computation and output layer projection—are particularly prominent. One of the methods is enabled by setting SGLANG_ENABLE_FP8_LM_HEAD=1, but it cannot be used together with the baseline option --enable-fp32-lm-head; NVIDIA points to the Ling cookbook notebook, which explains how to enable both together.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- MarkTechPostNVIDIA Shows 3 Ways to Speed Up MoE Inference on DGX Spark, Lifting Ling-3.0-flash Throughput 42.5%
NVIDIA's technical blog describes three stacked optimizations for running Ant Group inclusionAI's 124B-parameter Ling-3.0-flash (5.1B active) on a single DGX Spark at batch size one: Humming MXFP4 MoE kernels, an FP8 LM head, and built-in NEXTN speculative decoding. Across all 1,319 GSM8K questions, decode throughput rose 42.5% (37.81 to 53.89 tokens/s) and mean latency fell 24.5%, with accuracy essentially flat (96.89% to 96.82%). NVIDIA notes that combining the FP8 LM head with MTP requires a shared LM-head module, otherwise the framework silently falls back and throughput drops below the unoptimized baseline.
- NVIDIA Technical Blog3 Ways to Boost MoE Inference on NVIDIA DGX Spark
Profiling a batch-size-1 evaluation of Inclusion AI's Ling-3.0-flash on NVIDIA DGX Spark showed decoding was memory-bound, with model weight reads driving execution time. The MoE computation and the output-layer projection that produces scores across the model stood out as the two key operations.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 9Nov 4 07:00
- Change in the past 24 hours
- -51%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.