Challenges for human oversight in detecting AI-assisted diagnostic malfunctions
Challenges for human oversight in detecting AI-assisted diagnostic malfunctions
In a randomized experiment with 48 physicians classifying heart sounds while AI reliability declined across four blocks (90%, 60%, 50%, 0%), combined human-AI performance fell below unassisted human performance under low reliability, and only 12.5% of participants reported noticing the declining reliability. Cognitive forcing (requiring an independent classification before seeing the AI recommendation) did not improve diagnostic performance; at the lowest reliability block the AI-First group outperformed the AI-Second group. The authors conclude that human oversight alone may be insufficient to reliably detect AI system malfunctions and call for system- and organizational-level safeguards.
Source: Hacker News AI · nature.com