iCreatAI News
Skip to content
Google Developers Blog·· 2 days agoAI score58

Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText

Distributed LLM Training Across Metros: Streaming DiLoCo in MaxText

AI summary

Google describes DiLoCo, a distributed optimization algorithm that lets independent learner groups train locally for H steps before averaging weight deltas, and its Streaming DiLoCo variant, which partitions the model into P fragments and synchronizes one fragment per step to smooth bandwidth use. Benchmarks of Qwen3-32B on 2× TPU v5p-32 slices show cross-slice traffic dropping from 4,258.8 GB (data parallel) to 67.0 GB, with 3.37× and 9.57× speedups at 30 Gbit/s and 10 Gbit/s inter-slice throttling. Streaming DiLoCo is available in MaxText v0.2.5 on PyPI, with Docker images and GitHub main.

Source: Google Developers Blog · developers.googleblog.com