ChatGPT demonstrates modest accuracy when assigning BI-RADS scores for mammograms and breast ultrasound exams, according to research published October 30 in Clinical Imaging.
A team led by Marc Succi, MD, from Mass General Brigham in Boston found that two iterations of the large language model (LLM) could correctly assign BI-RADS scores in two out of every three cases, with better performance seen for BI-RADS 5 cases. The models achieved the lowest scores in assigning lower BI-RADS categories.
“These findings provide breast radiologists with a valuable foundation for understanding the current capabilities and limitations of off-the-shelf large LLMs in image interpretation,” Succi told AuntMinnie.com.
Previous reports suggest that large language models can correctly recommend appropriate imaging modalities for patients based on their clinical presentation. They can also correctly determine BI-RADS categories based on textual imaging reports, according to an earlier 2024 study.
Succi and colleagues conducted a pilot study that explored whether ChatGPT-4 and ChatGPT-4o, the latter of which adds multimodal processing, can assist with generating BI-RADS scores from mammographic and breast ultrasound images.
The team tested both models using 77 breast cancer images from radiopaedia.org and analyzed images in separate sessions to avoid bias.
Both ChatGPT-4 and ChatGPT-4o scored 66.2 % accuracy across all BI-RADS cases. However, this varied among BI-RADS categories. The models scored the highest when assessing BI-RADS 5 cases, 84.4% for GPT-4 and 88.9% for GPT-4o. Both models, however, scored 0% when assigning BI-RADS 3 categories and struggled with BI-RADS 1 and 2 categories.
“The models were able to handle high-risk cases effectively but tended to overestimate the severity of lower-risk cases,” Succi said.
Of the incorrect BI-RADS 1 to 3 grading for GPT-4 and GPT-4o, 64.2% and 76.4% were two grades higher than the correct score, respectively. When compared to the ground truth, the models achieved an interrater agreement of 0.72 and 0.68, respectively.
Finally, both models achieved higher accuracy for mammograms at 67.6% compared with 55.6 % for ultrasound images.
Succi said that the subtle differences in lower-risk cases may be harder for the LLMs to distinguish.
“Additionally, the models might have been trained on datasets that contain more high-risk cases, potentially influencing their accuracy,” he added.
Succi said that the research team is committed to discovering ways that LLMs can effectively support clinicians, with current projects spanning a range of applications, both in and outside of radiology.
“We're particularly interested in applications of AI for patient triage and patient education,” he told AuntMinnie.com.
The full study can be accessed here.
![Examples of ultrasound findings and techniques. (A) Images in a 39-year-old male patient with a mass in the left thigh. The mass is heterogeneous on the B-mode US image (compared with the patient in D) and showed increased microvascularity (superb microvascular imaging [SMI]) and shear-wave elastography (SWE) values. Undifferentiated pleomorphic sarcoma was diagnosed at biopsy (with pleomorphic rhabdomyosarcoma in surgical specimen). (B) Images in an 18-year-old male patient with a mass in the left leg. The mass is hypoechoic on the B-mode image, with no other findings suggestive of malignancy. The lesion is in contact with the cortex of the tibia, which is slightly irregular. CT revealed a doubtful anteromedial tibial erosion. The microvascular study demonstrated high vascularization, suggestive of malignancy. Periosteal Ewing sarcoma was diagnosed with both histologic and immunohistochemical confirmation. (C) Images in a 69-year-old female patient with a lump growing on the outside of the left leg. Multiple SWE examinations were performed (please note the high values obtained in the measurements, whereas the color map highlights the stiffness relative to adjacent tissues). SMI showed areas of increased vascularization to target for sampling. Undifferentiated spindle cell sarcoma was diagnosed at biopsy, with residual leiomyosarcoma in the surgical specimen after neoadjuvant therapy. (D) Images in a 56-year-old female patient with a mass in the right thigh. The mass is heterogeneous at both B-mode ultrasound (similar to patient A) and MRI (coronal T2-weighted spectral attenuated inversion recovery [SPAIR]; T1-weighted pre-contrast and postcontrast imaging), which even shows uptake after the administration of paramagnetic contrast material, which is traditionally suggestive of malignancy. Low values at SMI and elastography are suggestive of benignity. Spindle cell lipoma was diagnosed at biopsy, with atypical spindle cell lipomatous tumor in the surgical specimen.](https://img.auntminnie.com/mindful/smg/workspaces/default/uploads/2026/08/images-radiol250278fig2.APCFLSvX6p.jpg?auto=format%2Ccompress&dpr=2&fit=crop&h=167&q=70&w=250)
