EUROPA improves 1T MoE training throughput on JUPITER
Understand this story
The EUROPA consortium (composed of Domyn, Fraunhofer IAIS and IIS) says it used PyTorch and TorchTitan to pretrain an approximately 1T-parameter MoE on the EuroHPC JUPITER supercomputer, raising per-GPU throughput from 90 tokens/s to about 1,300 tokens/s. JUPITER is a publicly funded research infrastructure at Forschungszentrum Jülich, shared by hundreds of projects. EUROPA says it was selected to develop an open European MoE model with at least 400B parameters, supporting all 24 official EU languages and strictly complying with the EU AI Act. EUROPA says that to make results reproducible by others, it has submitted all fixes upstream to PyTorch and TorchTitan.
AI overview based on source reports · Updated 2 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- PyTorch BlogTraining a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan
The EUROPA consortium (Domyn, Fraunhofer IAIS and IIS) describes scaling pretraining of a ~951.9B-parameter MoE model (32.8B active per token) on JUPITER, the EuroHPC exascale supercomputer at Forschungszentrum Jülich, raising throughput from 90 to ~1,300 tokens/sec per GPU across 384 NVIDIA GH200 GPUs per DP replica. Key optimizations built on PyTorch and TorchTitan include FlexShard placement for the Muon optimizer, activation-checkpoint offloading to the Grace CPU, and running FSDP's all-gather setup on the compute stream via unshard_async_op, freeing ~14 GiB of HBM per GPU. The fixes were upstreamed to PyTorch and TorchTitan, with training reaching ~1,300 tokens/s per GPU on 1,536 GPUs across 4 DDP replicas.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 9Nov 12 22:00
- Change in the past 24 hours
- -50%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.