XYBench: Helping with the problem behind the question

XYBench is a new benchmark of 8,115 real-world questions, drawn from Stack Overflow, Stack Exchange, curated examples, and WikiHow, that tests whether AI assistants recognize when a user's request rests on a misconception (the "XY problem") and redirect them toward their actual goal. Evaluating Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, the authors found every model was more likely to address the stated request than the underlying goal, and all trailed human reference answers on misconception identification. The paper won the Best Resource Paper Award at EMNLP 2026.