HyperPod training loop adds silent data corruption detection
Understand this story
AWS published a technical blog introducing three checks added to the PyTorch DDP training loop on Amazon SageMaker HyperPod to detect silent data corruption (SDC): non-finite gradient checks, ring replay auditing, and weight fingerprint comparison. After a suspicious GPU is identified, HyperPod replaces the node and recovers from a checkpoint. According to the blog, reproducing this solution requires a HyperPod cluster orchestrated by Amazon EKS with at least three GPU nodes, and the complete code is in a companion GitHub repository. The blog also reminds that replacing a node destroys its root volume and secondary volumes, including data on local NVMe; after testing, the created resources should be deleted to avoid ongoing charges.
AI overview based on source reports · Updated 3 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogDetecting and recovering from silent data corruption on Amazon SageMaker HyperPod
This post describes adding three checks to a PyTorch DDP training loop to detect silent data corruption (SDC), where hardware such as a GPU returns a wrong but plausible number without raising an error: a non-finite gradient check, a ring replay audit that recomputes another node's micro-batch and compares loss bits and gradient fingerprints, and a weight fingerprint comparison after the optimizer step. When a check fires, the suspect rank is mapped to its node via the Kubernetes node name, the node is cordoned, and SageMaker HyperPod replaces it, with training resuming from a checkpoint saved before the suspect window. The checks were tested with software-injected faults on a HyperPod cluster with 4 ml.p5.48xlarge instances (32 NVIDIA H100 GPUs) and repeated on three instances, with full code in an accompanying GitHub repository.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 12 09:00
- Change in the past 24 hours
- -51%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.