MaxText implements Streaming DiLoCo for cross-city training
Understand this story
The Google DeepMind and Google Research teams have implemented Streaming DiLoCo in the open-source JAX training framework MaxText for training large language models over low-bandwidth wide area networks. The approach splits model parameters into P shards and synchronizes only one shard per step, reducing the communication required for cross-city training. The implementation ships with MaxText v0.2.5 and is available via PyPI, the corresponding Docker image, and GitHub main. The team recommends setting both the number of shards num_diloco_fragments (P) and the sync period diloco_sync_period (H) to N_layers + 1, so that exactly one shard is synchronized per training step.
AI overview based on source reports · Updated 17 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- Google Developers BlogDistributed LLM Training Across Metros: Streaming DiLoCo in MaxText
Google describes DiLoCo, a distributed optimization algorithm that lets independent learner groups train locally for H steps before averaging weight deltas, and its Streaming DiLoCo variant, which partitions the model into P fragments and synchronizes one fragment per step to smooth bandwidth use. Benchmarks of Qwen3-32B on 2× TPU v5p-32 slices show cross-slice traffic dropping from 4,258.8 GB (data parallel) to 67.0 GB, with 3.37× and 9.57× speedups at 30 Gbit/s and 10 Gbit/s inter-slice throttling. Streaming DiLoCo is available in MaxText v0.2.5 on PyPI, with Docker images and GitHub main.
Activity for this story
- Current comparable activity
- 4
- Peak comparable activity
- 6Nov 20 03:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.