Deep Learning-Based Multi-Modal Data Integration Enhancing Breast Cancer Disease-Free Survival Prediction

Information Science Artificial Intelligence Computer Science Medicine Oncology
breast cancer multi-modality deep learning pathological disease-free survival

Breast cancer is one of the most common malignancies among women worldwide. Although early intervention and appropriate treatment have significantly improved patient survival rates, approximately 30% of cases still experience recurrence and distant metastasis, resulting in a 5-year survival rate of less than 23%. Traditional clinical prediction methods, such as biomarkers, clinical imaging, and molecular testing, while valuable, have limitations including low sensitivity, high costs, limited availability, and intra-patient heterogeneity. Therefore, there is an urgent need to develop new methods to reliably predict the risk of recurrence and survival rates in postoperative breast cancer patients, enabling timely intervention and improving overall prognosis.

In recent years, the rapid development of artificial intelligence (AI) technology has provided new possibilities for breast cancer prognosis prediction. Deep learning, as a powerful AI technique, can extract valuable information from complex multimodal data, combining pathological images, molecular data, and clinical information to significantly improve the accuracy of predicting disease-free survival (DFS) in breast cancer. However, most existing studies are limited to single-modal data, lacking integrated analysis of multimodal data. Thus, how to effectively integrate multimodal data and develop high-precision prediction models has become a significant challenge in current breast cancer research.

Source of the Paper

This study was conducted by a research team including Zehua Wang, Ruichong Lin, Yanchun Li, and others from institutions such as Beijing Normal University-Hong Kong Baptist University United International College, Macau University of Science and Technology, and Sun Yat-sen University. The paper was published on May 29, 2024, in the journal Precision Clinical Medicine, titled “Deep learning-based multi-modal data integration enhancing breast cancer disease-free survival prediction.”

Research Process and Results

1. Data Collection and Preprocessing

The research team retrospectively collected pathological images, molecular data, and clinical data from The Cancer Genome Atlas (TCGA) and an independent institution in China. The study included 1,020 non-metastatic breast cancer patients, divided into a training cohort (n=741), an internal validation cohort (n=184), and an external testing cohort (n=95). All patients provided preoperative pathological images and were rigorously screened based on inclusion and exclusion criteria.

During the data preprocessing phase, the research team enhanced the quality of pathological images and performed segmentation. All pathological images were scanned at 20x magnification and processed using a KF-PRO-005-EX digital pathology scanner. The images were segmented into 256×256 pixel patches, and feature extraction was performed using the ResNet50 model to generate 1024-dimensional feature vectors.

2. Molecular Data Preprocessing

To ensure the quality and reliability of molecular data, the research team standardized the gene expression information of 741 patients in the training cohort. Through univariate Cox regression analysis, 219 genes significantly associated with prognosis were identified. Additionally, the team used the XCell tool to analyze immune cell data from 96 patients, quantifying the expression profiles of 64 immune and stromal cell types.

3. Development and Training of the Deep Learning Model

The research team developed a deep learning model based on multi-instance learning, named DeepClinMed-PGM (Deep Learning Clinical Medicine based Pathological Gene Multi-modal model). This model integrates pathological images, molecular data, and clinical information to predict patients’ DFS.

In the feature extraction stage, the model extracted features from pathological image patches using ResNet50 and weighted the features through a self-attention mechanism. In the survival prediction stage, the model integrated pathological image features, molecular data, and clinical information into a fully connected layer, ultimately outputting the patient’s DFS risk score.

4. Model Performance Evaluation

The research team evaluated the DeepClinMed-PGM model in the training cohort, internal validation cohort, and external testing cohort. The results showed that the model achieved AUC values of 0.979, 0.957, and 0.871 for 1-year, 3-year, and 5-year DFS predictions, respectively, in the training cohort; 0.886, 0.745, and 0.825 in the internal validation cohort; and 0.851, 0.878, and 0.938 in the external testing cohort. Additionally, the model’s C-index values in the three cohorts were 0.925, 0.823, and 0.864, demonstrating high predictive accuracy.

Through Kaplan-Meier analysis, the research team further validated the model’s risk stratification capability. In the training cohort, there was a significant difference in DFS between the high-risk and low-risk groups (HR=0.027, 95% CI: 0.0016–0.046, p<0.0001). This trend was also confirmed in the internal validation cohort and external testing cohort.

5. Model Visualization and Interpretation

To gain a deeper understanding of the model’s prediction mechanism, the research team used the Grad-CAM algorithm to visualize key regions in pathological images. Through the generated heatmaps, researchers were able to identify high-density regions in the tumor microenvironment that were closely related to patient prognosis. Additionally, the team conducted Gene Ontology (GO) and Kyoto Encyclopedia of Genes and Genomes (KEGG) pathway analyses, revealing differences in immune cell infiltration and gene expression between high-risk and low-risk groups.

Conclusion and Significance

The DeepClinMed-PGM model developed in this study significantly improves the accuracy of breast cancer DFS prediction by integrating pathological images, molecular data, and clinical information. The model not only demonstrated excellent predictive performance across multiple cohorts but also provided clinicians with deeper insights through visualization techniques, aiding in the formulation of personalized treatment plans.

The scientific value of this study lies in its pioneering integration of multimodal data into a deep learning framework, offering a new method for breast cancer prognosis prediction. Its practical value lies in helping clinicians more accurately assess patients’ recurrence risks, thereby developing more effective treatment strategies. Furthermore, this study provides an important reference for future development of prognosis prediction models for other cancers.

Research Highlights

  1. Multimodal Data Integration: First to integrate pathological images, molecular data, and clinical information into a deep learning model, significantly improving prediction accuracy.
  2. High Predictive Performance: Validated the model’s high predictive performance across multiple cohorts, with AUC values and C-index demonstrating excellent predictive capability.
  3. Visualization and Interpretability: Provided deep insights into the model’s prediction mechanism through the Grad-CAM algorithm and heatmap analysis, enhancing model interpretability.
  4. Personalized Treatment: Enabled personalized treatment plans through risk stratification, improving patient survival rates.

Other Valuable Information

The research team also found significant differences in immune cell infiltration and gene expression between high-risk and low-risk groups, offering new clues for future research on breast cancer immunotherapy. Additionally, the team plans to apply this model to other types of cancer to validate its generalizability and scalability.

This study, through the integration of multimodal data using deep learning techniques, provides a new method for breast cancer prognosis prediction, holding significant scientific and practical value.