AWS launches Model Leaderboard in SageMaker Studio
Understand this story
AWS has launched the JumpStart Model Leaderboard within SageMaker Studio, letting users compare open-weight models side by side on quality, speed, and cost on the same page. AWS says that at launch it covers more than 70 models from providers including Meta, Google, Qwen, IBM, and Arcee, with more to be added over time. Using the leaderboard requires SageMaker Studio access and permission to deploy JumpStart models. Not all JumpStart models have benchmark scores yet, and the first batch of benchmarks focuses mainly on large language models. The cost estimates shown on the page reflect only SageMaker instance costs, calculated at the on-demand hourly rate for the default instance type of the selected configuration.
AI overview based on source reports · Updated 16 hr ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogIntroducing the Amazon SageMaker JumpStart Model Leaderboard
AWS launched the SageMaker JumpStart Model Leaderboard, a view in SageMaker Studio that compares 70+ open-weight models from providers including Meta, Google, Qwen, IBM, and Arcee on quality, speed, and cost. Quality scores come from three benchmarks (BIG-Bench Hard, GPQA Diamond, and MMLU-Pro) run by AWS with the same evaluation setup for every model, while performance and cost metrics are measured by sending a standardized test workload to endpoints using each JumpStart deployment configuration. Users can filter models, select a use case profile and optimization goal, and then deploy or customize a model.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 19 06:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.