iCreatAI News
Skip to content
Trending storyUpdating

METR uses an investigation agent to measure alignment-evaluation cues

1 reports1 sources15 hr agoUpdated

Understand this story

AI overview

METR placed an investigative agent into 19 alignment evaluations and 36 capability evaluation environments, having it judge whether it was in an alignment evaluation, in order to identify which features let models recognize the type of evaluation. The results show that Opus 4.8 and Opus 5.5 could almost perfectly distinguish the two types of evaluations; under both direct scoring and open-ended responses, Opus 5.5 ranked every alignment evaluation above all capability evaluations.

AI overview based on source reports · Updated 42 min ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 16
  1. METR Blog
    Measuring alignment eval tells

    Researchers ran investigator agents inside 19 alignment evals and 36 capability evals to see whether models can tell what kind of eval they are in, and found that investigator agents have become increasingly able to discriminate between the two, with Opus 4.8 and Opus 5.5 nearly perfect on their dataset. The agents cite environment-level tells (mocked services, simulated infrastructure, implausible scenarios) and task-level tells (planted shortcuts, impossible legitimate success, conflicting goals, scripted users), and in one honeypot built by modifying a capabilities eval, Opus 5.5 identified all four task-level modifications. The authors frame this as understanding whether such tells exist, not as measuring whether they causally affect agent behavior during the eval.

Activity for this story

Not enough continuous observations to show a trend yet.