Combining Human Expertise with Artificial Intelligence: Experimental Evidence from Radiology
Nikhil Agarwal, Alex Moehring, Pranav Rajpurkar, Tobias Salz
NBER Working Paper Series · 2023
Abstract
Examines how expert radiologists combine their own judgment with recommendations from an AI model. The authors find that radiologists treat AI predictions as independent information and underweight them, which often leads to collaborative results that underperform the AI model alone.
Methodology
A randomized trial where radiologists evaluated chest X-rays. Radiologists in the treatment group were shown AI-generated probability scores of pathologies, and their diagnostic accuracy was compared to both human-alone and AI-alone baselines.
Findings
Radiologists consistently underweighted AI inputs, treating them as if they were uncorrelated with their own signals. This correlation neglect and conservatism meant that the joint human-AI diagnostic accuracy was lower than the AI model working in isolation.
Applied nudge
Design joint human-AI workflows so that experts are nudged to treat the AI's inputs not as separate opinions, but as integrated priors or structured checklists.
Citation
Agarwal, N., Moehring, A., Rajpurkar, P., & Salz, T. (2023). Combining human expertise with artificial intelligence: Experimental evidence from radiology. National Bureau of Economic Research, Working Paper No. 31422.
↗ Download PDF