SageMaker HyperPod adds GPU training observability
Understand this story
AWS has introduced enhanced observability capabilities for Amazon SageMaker HyperPod to diagnose GPU training issues faster. Through CloudWatch's DetectGroupOutliers API, the capability compares utilization, memory, temperature, power, and SM clock frequency across GPUs in the same training job, flags anomalous GPUs, and uses static diagnostic rules to identify common failures such as XID errors, ECC memory errors, EFA throughput drops, and thermal throttling. Default metrics are enabled automatically and cover cluster health, CPU/disk/OS utilization, GPU utilization and temperature, and per-job resource usage; advanced metrics must be actively enabled by the user and provide finer-grained host-level metrics such as the full node exporter suite, incurring costs because they increase the volume of telemetry data sent to the metrics store. The capability is available in all AWS Regions where SageMaker HyperPod is offered.
AI overview based on source reports · Updated 18 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogIntroducing enhanced infrastructure observability in Amazon SageMaker HyperPod
Amazon SageMaker HyperPod now offers enhanced observability that combines diagnostic summaries, using CloudWatch's DetectGroupOutliers API to flag GPUs deviating from peers on utilization, memory, temperature, power, and SM clock, with static diagnostic rules for hardware faults, EFA throughput drops, throttling, and overheating. The capability works across EKS and Slurm orchestrators, collects default metrics every 30 seconds at roughly 1–2% CPU overhead, and offers opt-in advanced metrics including the full node exporter suite, NVIDIA DCGM and AWS Neuron metrics, and EFA metrics, with CloudWatch as the new default metric store alongside Amazon Managed Service for Prometheus. It is delivered as an add-on for EKS clusters and an extension for Slurm clusters, and is available today in all AWS Regions where SageMaker HyperPod is available.
Activity for this story
- Current comparable activity
- 6
- Peak comparable activity
- 10Nov 20 02:00
- Change in the past 24 hours
- –
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.