AI Bot Passes UK Radiology Exam

An AI system recently passed a radiology exam with an average accuracy of 79.5% across 10 tests, comparable to the average accuracy of human radiologists at 84.8%, researchers in the UK found.

The researchers, who published their study in British Medical Journal, wanted to determine whether an AI bot could pass the “quick report” part of the Fellowship of the Royal College of Radiologists (FRCD) exam, which radiologists in the UK must pass to complete their training. The Quick Report section of the three-part exam requires candidates to interpret 30 radiographs in 35 minutes. To pass, candidates must report at least 90% of radiographs correctly. The AI ​​scored over 90% on two of the 10 tests it took.

“This part of the exam is designed to ‘stress test’ candidates for speed and accuracy, providing a mix of challenging common and unusual cases commonly encountered in clinical practice by the general and emergency department are referred for radiological interpretation,” the researchers said.

The researchers used 10 simulated FRCR Rapid Report tests for the study. The AI ​​candidate was a Smartergence tool developed by the French AI company Millview which is commercially available. The study also included 26 radiologists who had taken and passed the actual FRCR exam within the past year.

When no non-interpretable images were used, which were not recorded by the AI ​​at all, the AI ​​candidate had an average overall accuracy of 79.5% and was able to pass two out of 10 mock tests. Radiologists were 84.8% accurate and were able to pass four out of 10 mock exams on an average.

AI diagnosed images that radiologists typically diagnose correctly 91% of the time. For images that were misdiagnosed by most radiologists, the AI ​​was wrong only 50% of the time; Correct diagnosis of hands, carpal bones and feet.

The AI ​​candidate needs more training analyzing areas that are considered ‘non-descriptive’ such as the abdomen and axial skeleton.

While the AI ​​candidate had “relatively high” accuracy, it only scored the highest on one mock test.

“The AI ​​candidate will still need further training to achieve the same level of performance and skill as the average FRCR newly qualified radiologist,” the researchers said, adding that they still need training on “subtle musculoskeletal abnormalities.”