iCreatAI News
Skip to content
Trending storyWatching

METR study: agents can identify alignment evaluations

1 reports1 sources2 days agoUpdated

Understand this story

AI overview

METR placed investigative agents into 19 alignment evaluations and 36 capability evaluation environments, measuring whether they could tell which type of evaluation they were in. The results showed that Opus 4.8 and Opus 5.5 distinguished them nearly perfectly; for Opus 5.5, both direct scoring and open-ended scoring ranked every alignment evaluation above all capability evaluations. METR said the agents mainly judged based on cues such as the environment looking like a simulation or the task looking like a honeypot, and classified these cues into two categories: environment-level and task-level.

AI overview based on source reports · Updated 1 days ago

Coverage timeline

Follow the coverage to explore different perspectives.

Nov 16
  1. METR Blog
    Measuring alignment eval tells

    Anthropic researchers ran investigator agents inside 19 alignment evals and 36 capability evals to see whether models can tell what kind of eval they are in, using a direct 0–100 probability score and an LLM-judged open-ended answer. They report that investigator agents have become increasingly able to discriminate the two, with Opus 4.8 and Opus 5.5 nearly perfect (Opus 5.5 ranked every alignment eval above every capability eval), and that agents cite environment-level tells (simulated or contrived environments) and task-level tells (planted shortcuts, impossible legitimate success, conflicting goals, scripted users). The authors note interrogation can prime models and that they do not claim these beliefs causally affect agent behavior during the tasks.

Activity for this story

Current comparable activity
3
Peak comparable activity
7Nov 17 07:00
Change in the past 24 hours
-49%

Trends compare the same fully observed participants, which may cover fewer participants than current activity. Move the pointer or tap the chart to inspect hourly activity; use the left and right arrow keys with a keyboard.