Tuesday, December 1 | 3:00 p.m.-4:00 p.m. | T7-SSIN03-8 | Room E353A
Multimodal large language models can generate accurate differential diagnoses and may support clinical decision-making, according to this presentation.
Multimodal artificial intelligence (AI) models may be better at reading radiologists’ descriptions than at reading the images themselves. That’s the takeaway from a study that put three large language models (LLMs) to the test on radiology teaching cases.
Presenter Julius Holzschuh of German Cancer Research Center and colleagues evaluated the open-source medical foundation model MedGemma-27B and the general-purpose models GPT-5 and GPT-4.1 on 280 teaching cases spanning multiple subspecialties.
Each model was tested under four conditions: images alone, images plus a brief clinical history, images plus history plus a written description of the key findings, and history plus written findings without images. The researchers assessed how often the models identified the key findings, offered correct differential diagnoses, and reached the correct final diagnosis.
From images alone, final diagnosis accuracy was low: 20% for GPT-5, about 14% for GPT-4.1, and about 1% for MedGemma. Adding a written description of the findings produced the biggest gain for every model, raising those figures to about 75% , 63%, and 41%, respectively. Adding images to a case already described in text, however, didn’t significantly help, indicating that current vision components add little if a radiologist’s description of the findings is available, according to the researchers. Overall, GPT-5 performed best.
“Given a description of the key imaging findings, multimodal large language models can generate accurate differential diagnoses and may support clinical decision-making, but they are not yet reliable enough to interpret images on their own,” the researchers wrote.
Attend this Tuesday afternoon talk to learn more.
