RETFound: A Generalizable Foundation Model to Improve Ocular Disease Diagnosis using a Combination of SSL and SL-ImageNet

RETFound enhances the performance of detecting ocular diseases by learning to identify disease-related lesions. Hard exudates and haemorrhages are some of the pathological patterns that can be used to diagnose ocular diseases. These features have abnormal variations in structure and colour which can be seen by the eye. RETFound can identify disease-related patterns and correctly diagnose ocular diseases (for example, myopia and diabetic retinopathy cases in Extended Data Fig. 6b). RETFound is first in several tasks, followed by sl-imagenet. SL-ImageNet pretrains the model using supervised learning on ImageNet-21k, which contains 14 million images with 21,000 categories of natural objects with diverse shapes and textures, such as zebras and oranges. The model can learn low-level features such as lines, curves and edges to identify the boundary of abnormal patterns if they are diverse. In this paper, we demonstrate that by using SSL successively on natural images and unlabelled retinal images, a generalizable foundation model (RETFound) can be developed to further improve ocular disease diagnosis and prognosis, even outperforming the powerful SL-ImageNet.

The method used by scientists was similar to one used to train large-language models. The tool learns how to pick the next word from human-generated text in the context of the preceding words. RETFound learns how to predict what missing portions of pictures should look like by using a lot of Retinal photos.

“Over the course of millions of images, the model somehow learns what a retina looks like and what all the features of a retina are,” says Pearse Keane, an ophthalmologist at Moorfields Eye Hospital NHS Foundation Trust in London who co-authored a paper published today in Nature1 describing the tool. This forms the cornerstone of the model, and classifies it as what some call a foundation model, which means that it can be adapted for many tasks.

A person’s retinas can offer a window into their health, because they are the only part of the human body through which the capillary network, made up of the smallest blood vessels, can be observed directly. “If you have some systemic cardiovascular disease, like hypertension, which is affecting potentially every blood vessel in your body, we can directly visualize [that] in retinal images,” Keane says.

Retinal Disease Detection Using Multidimensional Unlabelled Patient Data: A Foundation Model for Efficient and Safe Adaptation

Using unlabelled data to initially train the model “unblocks a major bottleneck for researchers”, says Xiaoxuan Liu, a clinical researcher who studies responsible innovation in AI at the University of Birmingham, UK. The director of the Center for Artificial Intelligence in Medicine and Illustrates is also a radiologist. “High-quality labels for medical data are extremely expensive, so label efficiency has become the coin of the realm,” he says.

RETFound may be the basis for using other types of medical scans. “It will be interesting to see whether these methods generalize to more complex images,” Langlotz says — for example, to magnetic resonance images or computed tomography scans, which are often three- or even four-dimensional.

The authors have made the model publicly available, and hope that groups around the world will be able to adapt and train it to work for their own patient populations and medical settings. They could use data from their country to fine-tune it and have something that is more specialized for their use.

Liu says, “This is tremendously exciting.” The risk comes from using RETFound as the basis for other models. Future models that were built from it could leak any limitations embedded in the tool. “It is now up to the authors of RETFound to ensure its ethical and safe usage, including transparent communication of its limitations, so that it can be a true community asset.”

This work introduces a new foundation model called RETFound, and looks at its generalizability in adapting to different tasks. After training on large-scale unlabelled retinal images using an advanced SSL technique (masked autoencoder), RETFound can be efficiently adapted to a broad range of disease detection tasks, resulting in significant performance improvements for detecting ocular diseases and predicting cardiovascular and neurodegenerative diseases. It is a medical foundation model that has been developed and assessed, and shows considerable promise for leveraging such multidimensional data without constraints of enormous high-quality labels.

There is a fall in performance when models are tested against new groups that have differing demographic profiles and on the devices that were used. The external evaluation of ocular disease diagnosis has observed this phenomenon in the past. 3b). For example, the performance on ischaemic stroke drops (RETFound’s AUROC decreases by 0.16 with CFP and 0.19 with OCT). In the challenging oculomic tasks, the age and ethnicity profile of the internal and external validation cohorts (MEH-AlzEye and UK Biobank) as well as the imaging devices are significantly different (Supplementary Table 2), and this is likely to be reflected in the drop in performance when externally evaluated in the UK Biobank cohort. Compared to other models, RETFound achieves significantly higher performance in external evaluation in most tasks (Fig. 3b) as well as different ethnicities (Extended Data Figs. 9–11), showing good generalizability.