Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan

The EUROPA consortium (Domyn, Fraunhofer IAIS and IIS) describes scaling pretraining of a ~951.9B-parameter MoE model (32.8B active per token) on JUPITER, the EuroHPC exascale supercomputer at Forschungszentrum Jülich, raising throughput from 90 to ~1,300 tokens/sec per GPU across 384 NVIDIA GH200 GPUs per DP replica. Key optimizations built on PyTorch and TorchTitan include FlexShard placement for the Muon optimizer, activation-checkpoint offloading to the Grace CPU, and running FSDP's all-gather setup on the compute stream via unshard_async_op, freeing ~14 GiB of HBM per GPU. The fixes were upstreamed to PyTorch and TorchTitan, with training reaching ~1,300 tokens/s per GPU on 1,536 GPUs across 4 DDP replicas.