Apollo is a multimodal foundation model developed by Harvard researchers that combines data from radiology, pathology, genomics, and clinical records to predict disease risk and patient outcomes. Trained on 25 billion records from 7.2 million patients, Apollo outperforms 26 established clinical risk scores and can predict disease onset up to five years in advance with significantly higher accuracy.
- Apollo achieved an AUC of 0.79 for five-year breast cancer metastatic spread prediction, compared to baseline AUC of 0.64
- Trained on 25 billion records from 7.2 million patients across 28 medical modalities and 12 major specialties
- Apollo outperformed 26 established clinical risk scores, including Fib-4 and the AHA's PREVENT score
- FDA collaboration aims to reduce placebo patients needed in trials by 25%
- New clinical system one model provides rapid predictions without requiring large language model queries
PHILADELPHIA -- Multidisciplinary approaches with large data from radiology and pathology may be the future for large language models (LLMs), and one example was highlighted at the Society for Imaging Informatics in Medicine (SIIM)-CAIMI 2026 meeting.
In his keynote presentation, Faisal Mahmood, PhD, from Harvard Medical School and Brigham and Women's Hospital in Boston, outlined the work being done toward Apollo, a multimodal temporal foundation model.
“What we really want to get to is that can you ask any question and we would be able to answer it,” Mahmood said.
Mahmood’s research has focused on deep learning approaches toward improving accuracy in pathology. The work he and colleagues have done include developing instance learning-based frameworks to analyze large, complex images for tasks including identifying cancers of unknown primary origin, assessing heart transplant rejection through endomyocardial biopsies, and predicting treatment response.
One model he and colleagues have been working on in recent years is Apollo, which Mahmood said has been trained and evaluated on over three decades of longitudinal hospital records. This includes 25 billion records from 7.2 million patients in the Mass General Brigham health system, representing 28 distinct medical modalities and 12 major medical specialties, radiology included.
Apollo takes data from radiology, pathology, genomics, and clinical data to create a single, temporally aligned representation of a patient’s condition. This includes risk prediction, similarity searching, statistical analysis, treatment response, and more.
Faisal Mahmood, PhD, shares insights from his team's work in developing an LLM framework that takes data from radiology, pathology, and other specialties to predict disease risk and monitor patient health.
Mahmood and colleagues in April had a study published showing how Apollo has potential in clinical forecasting. They created 322 prognosis and retrieval tasks from a held-out test set of 1.4 million patients. These tasks included predicting new disease onset risk up to five years in advance, disease progression, treatment response, risk of treatment-related adverse events, and hospital operations endpoints.
Apollo led to higher area under the curve (AUC) values over baseline scores for oncology prediction across the disease course. For example, five-year risk prediction of breast cancer metastatic spread through Apollo led to an AUC of 0.79 compared to the baseline AUC of 0.64. For two-year lung cancer prediction, Apollo achieved an AUC of 0.76 compared to the baseline AUC of 0.62.
Apollo outperformed 26 established clinical risk scores in the study, such as Fib-4 for measuring liver fibrosis risk and the American Heart Association’s Predicting Risk of cardiovascular disease EVENTs (PREVENT) score, Mahmood highlighted.
“You have all this digital data about a patient; you collapse it all down into a single vector that represents the patient at that given point in time,” Mahmood said.
Mahmood also highlighted the Apollo team is working with the U.S. Food and Drug Administration to evaluate how this approach can reduce the number of placebo patients needed to power a trial by about 25%.
The team recently introduced what they call a clinical “system one” model. This is a fast, lightweight decision model that pre-computes answers to common clinical questions across thousands of medical notes, providing rapid probabilistic outputs without the computational cost of querying a large language model from scratch.
Looking further ahead, the group is developing tools for 3D pathology, imaging tissue volumes using micro-CT scanners and open-top light sheet microscopes. The team is also exploring computational methods to impute spatial transcriptomics in 3D, potentially opening new avenues for drug target validation and biomarker discovery, Mahmood said.
“Once we have the embedding available, all these applications become a variety. It just makes the entire patient record much more computable,” he said. “So, instead of working in individual modalities and domains, you're working with a single representation that you can use for all these different things. And while developing all of this, we have individual representations come from each one of those 28 modalities; those could also be used.”
