iCreatAI News
Skip to content
Trending storyWatching

MaxText implements Streaming DiLoCo for cross-city training

1 reports1 sources2 days agoUpdated

Understand this story

AI overview

The Google DeepMind and Google Research teams have implemented Streaming DiLoCo in the open-source JAX training framework MaxText for training large language models over low-bandwidth wide area networks. The approach splits model parameters into P shards and synchronizes only one shard per step, reducing the communication required for cross-city training. The implementation ships with MaxText v0.2.5 and is available via PyPI, the corresponding Docker image, and GitHub main. The team recommends setting both the number of shards num_diloco_fragments (P) and the sync period diloco_sync_period (H) to N_layers + 1, so that exactly one shard is synchronized per training step.

AI overview based on source reports · Updated 17 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 19
  1. Google Developers Blog
    Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText

    Google describes DiLoCo, a distributed optimization algorithm that lets independent learner groups train locally for H steps before averaging weight deltas, and its Streaming DiLoCo variant, which partitions the model into P fragments and synchronizes one fragment per step to smooth bandwidth use. Benchmarks of Qwen3-32B on 2× TPU v5p-32 slices show cross-slice traffic dropping from 4,258.8 GB (data parallel) to 67.0 GB, with 3.37× and 9.57× speedups at 30 Gbit/s and 10 Gbit/s inter-slice throttling. Streaming DiLoCo is available in MaxText v0.2.5 on PyPI, with Docker images and GitHub main.

Activity for this story

Current comparable activity
4
Peak comparable activity
6Nov 20 03:00
Change in the past 24 hours
–

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.