AI Diagnostic Tool Underperforms in Detecting COVID-19 on Chest Radiographs

A prospective observational study published in Radiology: Artificial Intelligence reports that an interpretable AI diagnostic model for COVID-19 using chest radiographs was less accurate than radiologists reading the same exam.

Researchers at the University of Minnesota examined the real-time performance of an interpretable AI diagnostic model for COVID-19 that was implemented across 12 hospitals in the state. The study, supported by an AI for Health COVID-19 grant from Microsoft, found that the AI diagnostic tool achieved an area under the receiver operating characteristic curve of 0.70 on real-time validation. Further, the AI diagnostic system had a significantly lower accuracy at 63.5% compared with two radiologists who had 67.8% and 68.6% accuracy.

Participants who were positive for COVID-19 had had higher COVID-19 diagnostic scores than participants negative for COVID-19 (median, 0.1 [IQR, 0.0–0.8] vs 0.0 [IQR, 0.0–0.1], respectively; P , .001). Model performance remained unchanged over the study duration of 19 weeks. Model sensitivity was higher in men than women and higher in Asian and Black than White participants. However, the AI model showed improved diagnostic capabilities in all participants with more severe disease.

The authors surmise that “it may be impossible to develop a model solely based on chest radiograph findings to differentiate between participants with COVID-19 and non–COVID-19 diagnoses. This was an early hypothesis by our clinical content experts, as the radiographic appearance of COVID-19 is heterogeneous. The radiographic appearance of COVID-19 may range from no or minimal observable pathologic abnormality to severe acute respiratory distress syndrome and can progress and defervesce depending on the time of exposure and stage of disease.”

In summary, the authors note that AI may be useful as an adjunct in clinical decision-making concerning COVID-19 diagnosis, however, it is not a replacement tool for radiologists.