Readers of varying skill levels perform significantly better when they use computer-aided detection (CAD) to find lung nodules on CT scans, according to a new study from Stanford University in Stanford, CA.
Although CAD tends to equalize performance among readers of varying skills, readers perform differently based not only on experience levels and the presentation order of candidate nodules, but on the total time spent reading a case, according to a study published online in European Radiology on September 16.
Most commercial CAD algorithms rely on fixed performance thresholds, but that doesn't mean the algorithm sees all nodule candidates in the same way, wrote Dr. Justus Roos and colleagues from Stanford's radiology department. When lower thresholds are used for lesion inclusion, sensitivity typically increases at the expense of rising false positives.
"To our knowledge, the balance between sensitivity and specificity at varying CAD performance thresholds and the impact of CAD output on the reliability and interaction time of human users have not been described," they wrote.
The researchers aimed to evaluate the variability with which radiologists use CAD to improve nodule detection, to determine how diagnostic performance changes as a function of the incremental use of CAD, and to look for temporal trends of radiologists' performance using CAD. Using a CAD system developed at the institution permitted the systematic alteration of performance thresholds and the recording of any performance changes resulting from the changes, Roos and colleagues wrote.
The database included 20 outpatients (15 men, five women; ages 15-91 years; mean age, 64) who were referred for clinical suspicion of pulmonary nodules. MDCT images were acquired using 1-mm collimation, 120 kVp, and 200-300 mA. Transverse 1.25-mm sections were reconstructed at 0.6-mm intervals. The output of a noncommercial CAD software application developed at the institution was characterized by location and CAD score, a numerical value approximating the likelihood of the detection being a true nodule.
"The loci with the 50 highest CAD scores were identified for each patient and sorted by descending CAD score," Roos and his team explained. "Thus, the CAD candidate with the highest likelihood of being a true pulmonary nodule was shown first to the reader during their CAD evaluation."
The high number of lesion candidates per case permitted the analysis of incremental benefits and temporal trends that affect performance; however, the detector output (true positives versus false detections) was not optimized.
The readers each performed two interpretations: first, a "free search" not limited by time, to detect all nodules 3 mm or larger and rate nodule confidence on a scale of 1 to 5 (5 = definitely a nodule), followed by the CAD-assisted search with the highest CAD scores presented first, with cases in random order. The CAD application removed nodule candidates that had already been identified during each reader's free search. A confidence level of 3 or more was used as a threshold for positivity.
Detection sensitivity: CAD versus no CAD
|
Readers 1-3 detected nodules with sensitivities of 59%, 57%, and 44%, respectively, during free search, which increased with CAD use to 67%, 82%, and 60%, respectively. The false-positive rate ranged from 0.6 to 2.1 per patient for free search and from 0.85 to 2.7 per patient with CAD.
The CAD algorithm detected 141 of 190 (74%) nodules. Of these, 18% (25 of 141) were not found by any readers during free search. In comparison, 14% (27 of 190) of the nodules were detected by at least one reader at the free search but not by CAD. And 14% of all nodules were either undetected (11.4%) or rejected (2.6%) by all three readers.
"The readers rejected [true-positive] CAD candidates heterogeneously, with readers 1 and 3 rejecting significantly more than reader 2 (p < 0.001)," the authors wrote. However, the readers behaved similarly in their strong rejection of most of the false-positive CAD marks (p ≥ 0.85).
The CAD evaluation sharply increased sensitivity at first --- by 14% -- with only a minimal increase in false positives over 100 seconds, followed by a flattening of the sensitivity increase to only 2%.
"This transition resulted from a greater prevalence of true positive (TP) versus FP detections at early CAD evaluation and not by a temporal change in readers' performance," they noted.
Although mean time spent analyzing true-positive (9.5 seconds ± 4.5 seconds) and false-negative (8.4 seconds ± 6.7 seconds) detections were similar, false-positive decisions required much more reading time (mean 14.4 seconds ± 8.7 seconds) than true-negative decisions (mean 4.7 seconds ± 1.3 seconds), Roos and colleagues wrote.
"Our findings confirm the potential of CAD to improve radiologists' diagnostic performance in the detection of pulmonary nodules on chest MDCT," they wrote. However, the magnitude of the assistance provided by CAD was "influenced by the quality of the reader interpretation of CAD output as well as by the inherent technical properties of the algorithm."
The diagnostic performance with CAD varied among readers and "did not follow the patterns of reader performance during free search," they wrote. "However, CAD has the potential to equalize performance among readers by reducing individual detection errors of lung nodules on chest CT."
Additionally, analyzing CAD-dependent temporal variations in reader performance may help establish optimum reading times for CAD interaction, they concluded.
By Eric Barnes
AuntMinnie.com staff writer
November 4, 2009
Related Reading
CARS report: New CAD tool follows lung nodules over time, June 30, 2009
New lung cancer staging guidelines reflect improved CT, survival data, June 1, 2009
CAD performs well in subsolid pulmonary nodules found with CT, May 15, 2009
Computer-aided system increases detection of early-stage lung cancer, May 4, 2009
Copyright © 2009 AuntMinnie.com
