iCreatAI News
Skip to content
Trending storyWatching

AWS launches Model Leaderboard in SageMaker Studio

1 reports1 sources2 days agoUpdated

Understand this story

AI overview

AWS has launched the JumpStart Model Leaderboard within SageMaker Studio, letting users compare open-weight models side by side on quality, speed, and cost on the same page. AWS says that at launch it covers more than 70 models from providers including Meta, Google, Qwen, IBM, and Arcee, with more to be added over time. Using the leaderboard requires SageMaker Studio access and permission to deploy JumpStart models. Not all JumpStart models have benchmark scores yet, and the first batch of benchmarks focuses mainly on large language models. The cost estimates shown on the page reflect only SageMaker instance costs, calculated at the on-demand hourly rate for the default instance type of the selected configuration.

AI overview based on source reports · Updated 16 hr ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 19
  1. AWS Machine Learning Blog
    Introducing the Amazon SageMaker JumpStart Model Leaderboard

    AWS launched the SageMaker JumpStart Model Leaderboard, a view in SageMaker Studio that compares 70+ open-weight models from providers including Meta, Google, Qwen, IBM, and Arcee on quality, speed, and cost. Quality scores come from three benchmarks (BIG-Bench Hard, GPQA Diamond, and MMLU-Pro) run by AWS with the same evaluation setup for every model, while performance and cost metrics are measured by sending a standardized test workload to endpoints using each JumpStart deployment configuration. Users can filter models, select a use case profile and optimization goal, and then deploy or customize a model.

Activity for this story

Current comparable activity
3
Peak comparable activity
10Nov 19 06:00
Change in the past 24 hours
-49%

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.