XYBench: Helping with the problem behind the question

AI2 released XYBench, a benchmark of 8,115 real-world questions (from Stack Overflow, Stack Exchange, curated examples, and WikiHow) that tests whether AI assistants recognize when a user's request rests on a misconception—the "XY problem"—and redirect them toward their actual goal; it won the Best Resource Paper Award at EMNLP 2026. Evaluating Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B, and DeepSeek-V4 Pro, the researchers found all five models were more likely to address the stated request than the underlying goal and trailed human reference answers on misconception identification. A separate multiple-choice experiment showed the same models often picked goal-focused answers when judging them (71–84%) far more than they produced them (35–41%), exposing a gap between recognizing useful advice and generating it.