Tuesday, December 1 | 3:00 p.m.-4:00 p.m. | T7-SSNPM02-4 | Room S503
Vendor-reported performance metrics don’t guarantee that a radiology AI device will perform well in clinical use, researchers will caution in this talk.
Although AI devices are typically evaluated on sensitivity and specificity, those measures don’t account for how common a disease is in practice, noted presenter Kelvin Stevens of the University of Freiburg, and colleagues. When prevalence is low, even an accurate device can produce far more false positives than true positives, a phenomenon known as the false-positive paradox.
The group reviewed U.S. Food and Drug Administration 510(k) summaries from 2024 and 2025 for 38 devices across 19 pathologies. They then applied real-world prevalence data to estimate how the devices would perform clinically.
Although the devices averaged more than 90% sensitivity and specificity, more than half of their positive flags were often false, especially for less common conditions. For one large vessel occlusion detector, about 71 of every 100 flags were predicted to be false positives, close to the 67% observed in a real-world case study. Only 10% of the summaries reported predictive values.
“Vendors should disclose sensitivity and specificity along with test set prevalence and Bayes-adjusted predictive metrics across multiple thresholds to enable local practices to anticipate AI error rates for their patient population,” the researchers wrote. “Radiologists should assess predictive values like the false-discovery rate in accordance with local prevalence rather than vendor-reported [positive predictive value], which overstates real-world performance.”
Sit in on this Tuesday afternoon presentation to get all of the details.
