iCreatAI News
Skip to content
AWS Machine Learning Blog· Roy Allela·· 2 days agoAI score36

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

AI summary

Amazon SageMaker HyperPod adds three capabilities to raise training goodput: the MultiNodeTrainingValidation deep health check, which runs a fixed Megatron-LM workload on each instance and then across instances to catch GPUs that pass hardware tests but produce NaN or infinite loss; the HyperPod Kubernetes Resource Monitoring Agent, which watches nodes, critical pods, DaemonSets, training jobs and opt-in NCCL RAS state on EKS clusters and reports problems as Kubernetes events and CloudWatch logs; and hung job detection for Slurm clusters, which flags jobs that stop making progress and can cancel them so Slurm requeues them. The post describes setup steps, prerequisites, sample commands and log output for each capability, and estimates that a 15-minute hung job detection rule on a 32-instance ml.p5.48xlarge job could cut a hang's cost from about $16,900 to about $530.

Source: AWS Machine Learning Blog · aws.amazon.com