Automated grounding verification of AI statements about chest radiographs
Automated grounding verification of AI statements about chest radiographs
The paper presents BRACE, an automated framework that builds training examples from public radiology reports and Radiopaedia articles to check a statement about a patient's image against both the report and the cited evidence. Across 42 verifiers (0.1–284B parameters) on five public chest radiograph datasets, verifiers trained with BRACE accepted a mean of 0.239 of AI-written statements contradicted by the report at a threshold accepting 0.90 of supported examples, versus 0.830 for verifiers trained without contradicted examples, and accepted 0.169 of statements with an agreeing report but unsupporting citation, versus 0.830 for zero-shot LLMs. Two board-certified radiologists' independent labels agreed with the framework's labels (k > 0.910) nearly as closely as with each other (k = 0.952), and a recipe food-allergen instantiation accepted 0.493 of contradicted statements versus 0.898 for a lexical baseline.
Source: Hacker News AI · arxiv.org