Detecting and recovering from silent data corruption on Amazon SageMaker HyperPod

This post describes adding three checks to a PyTorch DDP training loop to detect silent data corruption (SDC), where hardware such as a GPU returns a wrong but plausible number without raising an error: a non-finite gradient check, a ring replay audit that recomputes another node's micro-batch and compares loss bits and gradient fingerprints, and a weight fingerprint comparison after the optimizer step. When a check fires, the suspect rank is mapped to its node via the Kubernetes node name, the node is cordoned, and SageMaker HyperPod replaces it, with training resuming from a checkpoint saved before the suspect window. The checks were tested with software-injected faults on a HyperPod cluster with 4 ml.p5.48xlarge instances (32 NVIDIA H100 GPUs) and repeated on three instances, with full code in an accompanying GitHub repository.