AWS open-sources the UAEF agent evaluation framework
Understand this story
The AWS team has open-sourced the Universal Agent Evaluation Framework (UAEF), with the code released under the Apache 2.0 license in the awslabs/uaef GitHub repository. It uses adapters to unify execution traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore, and Langfuse into the AgentTrace format, then scores them using 36 built-in metrics across 7 dimensions. UAEF does not require ground-truth answers: only metrics that need comparison against reference answers use them, and such metrics are automatically skipped when ground truth is missing, so partially labeled datasets and unlabeled production traffic can still be evaluated. In addition, only persisted run records can be used for comparison.
AI overview based on source reports · Updated 3 days ago
Coverage timeline
Follow the coverage to explore different perspectives.
- AWS Machine Learning BlogFeaturedUAEF: A framework-agnostic system for closed-loop evaluation of AI agents
AWS introduces the Universal Agent Evaluation Framework (UAEF), an open-source, framework-agnostic system that uses adapters to normalize agent traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore, and Langfuse into a common AgentTrace format, then scores them with 36 built-in metrics across 7 quality dimensions. Results are stored per experiment in Amazon DynamoDB and Amazon S3, enabling comparison of agent versions, regression detection, and report and insight generation, with both a Python library and a low-code web UI sharing the same engine. The post also provides a playbook for choosing metrics and preparing datasets, plus runnable notebooks for each supported framework.
Activity for this story
- Current comparable activity
- 3
- Peak comparable activity
- 10Nov 12 00:00
- Change in the past 24 hours
- -49%
Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.