# Improving Colorectal Cancer Screening and Risk Assessment through Predictive Modeling on Medical Images and Records

arXiv: https://arxiv.org/abs/2410.09880 · HTML: https://mlsanity.com/paper/2410.09880/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 84 references that are in the map.
- First posted on arXiv 13 Oct 2024.

## Builds on (most cited first, 11 of 11)

- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Data Efficient and Weakly Supervised Computational Pathology on Whole Slide Images](https://mlsanity.com/paper/2004.09666/) (2020)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Whole Slide Images based Cancer Survival Prediction using Attention Guided Deep Multiple Instance Learning Networks](https://mlsanity.com/paper/2009.11169/) (2020)
- [Attention-Based Deep Neural Networks for Detection of Cancerous and Precancerous Esophagus Tissue on Histopathological Slides](https://mlsanity.com/paper/1811.08513/) (2018)
- [Deep-Learning for Classification of Colorectal Polyps on Whole-Slide Images](https://mlsanity.com/paper/1703.01550/) (2017)
- [Development and Evaluation of a Deep Neural Network for Histologic Classification of Renal Cell Carcinoma on Biopsy and Surgical Resection Slides](https://mlsanity.com/paper/2010.16380/) (2020)
- [MHAttnSurv: Multi-Head Attention for Survival Prediction Using Whole-Slide Pathology Images](https://mlsanity.com/paper/2110.11558/) (2021)
- [Masked Pre-Training of Transformers for Histology Image Analysis](https://mlsanity.com/paper/2304.07434/) (2023)
- [Automated detection of celiac disease on duodenal biopsy slides: a deep learning approach](https://mlsanity.com/paper/1901.11447/) (2019)
- [Fully transformer-based biomarker prediction from colorectal cancer histology: a large-scale multicentric study](https://mlsanity.com/paper/2301.09617/) (2023)

## Abstract (arXiv)

Colonoscopy screening effectively identifies and removes polyps before they progress to colorectal cancer (CRC), but current follow-up guidelines rely primarily on histopathological features, overlooking other important CRC risk factors. Variability in polyp characterization among pathologists also hinders consistent surveillance decisions. Advances in digital pathology and deep learning enable the integration of pathology slides and medical records for more accurate CRC risk prediction. Using data from the New Hampshire Colonoscopy Registry, including longitudinal follow-up, we adapted a transformer-based model for histopathology image analysis to predict 5-year CRC risk. We further explored multi-modal fusion strategies to combine clinical records with deep learning-derived image features. Training the model to predict intermediate clinical variables improved 5-year CRC risk prediction (AUC = 0.630) compared to direct prediction (AUC = 0.615, p = 0.013). Incorporating both imaging and non-imaging data, without requiring manual slide review, further improved performance (AUC = 0.674) compared to traditional features from colonoscopy and microscopy reports (AUC = 0.655, p = 0.001). These results highlight the value of integrating diverse data modalities with computational methods to enhance CRC risk stratification.