Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod
Introducing enhanced infrastructure observability in Amazon SageMaker HyperPod
Amazon SageMaker HyperPod now offers enhanced observability that combines diagnostic summaries, using CloudWatch's DetectGroupOutliers API to flag GPUs deviating from peers on utilization, memory, temperature, power, and SM clock, with static diagnostic rules for hardware faults, EFA throughput drops, throttling, and overheating. The capability works across EKS and Slurm orchestrators, collects default metrics every 30 seconds at roughly 1–2% CPU overhead, and offers opt-in advanced metrics including the full node exporter suite, NVIDIA DCGM and AWS Neuron metrics, and EFA metrics, with CloudWatch as the new default metric store alongside Amazon Managed Service for Prometheus. It is delivered as an add-on for EKS clusters and an extension for Slurm clusters, and is available today in all AWS Regions where SageMaker HyperPod is available.
Source: AWS Machine Learning Blog · aws.amazon.com