iCreatAI News
Skip to content
Trending storyWatching

SageMaker HyperPod adds workload recovery and infrastructure checks

2 reports1 sources2 days agoUpdated

Understand this story

AI overview

AWS introduced workload recovery and infrastructure check capabilities for Amazon SageMaker HyperPod, aiming to detect training failures earlier that hardware tests cannot catch. AWS said these checks can improve the goodput of training workloads. Specifically, there are three: MultiNodeTrainingValidation runs at instance provisioning or on demand to confirm GPUs can train properly; Hung job detection runs with each Slurm job to identify jobs that have stopped making progress, deployed as an agent on each node and off by default; and the HyperPod Kubernetes Resource Monitoring Agent continuously monitors the cluster and reports the pods and nodes involved when a job fails.

AI overview based on source reports · Updated 15 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 20
  1. AWS Machine Learning Blog
    Amazon SageMaker Inference: 2026 launches in review, part 2

    This second part of Amazon's 2026 SageMaker Inference review walks through 13 launches (numbered 14–23) covering inference recommendations in SageMaker AI Studio, the SageMaker Python SDK v3, and coding agents, plus LoRA adapter support and SLA-based concurrency search. It also details Inference Components in the console, NVIDIA MIG GPU partitioning, managed NVIDIA Dynamo on HyperPod Inference, KV-cache-aware routing, the HyperPod Inference Gateway, and Cross-Region Inference Routing. The post closes with continuous performance monitoring and in-place rolling model updates, illustrated through a hypothetical support-assistant team scaling to more models and Regions.

Nov 19
  1. AWS Machine Learning Blog
    Introducing workload recovery and expanded infrastructure checks on Amazon SageMaker HyperPod

    Amazon SageMaker HyperPod adds three capabilities to raise training goodput: the MultiNodeTrainingValidation deep health check, which runs a fixed Megatron-LM workload on each instance and then across instances to catch GPUs that pass hardware tests but produce NaN or infinite loss; the HyperPod Kubernetes Resource Monitoring Agent, which watches nodes, critical pods, DaemonSets, training jobs and opt-in NCCL RAS state on EKS clusters and reports problems as Kubernetes events and CloudWatch logs; and hung job detection for Slurm clusters, which flags jobs that stop making progress and can cancel them so Slurm requeues them. The post describes setup steps, prerequisites, sample commands and log output for each capability, and estimates that a 15-minute hung job detection rule on a 32-instance ml.p5.48xlarge job could cut a hang's cost from about $16,900 to about $530.

Activity for this story

Current comparable activity
4
Peak comparable activity
5Nov 20 05:00
Change in the past 24 hours
–

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.