# Streaming convolutional neural networks for end-to-end learning with multi-megapixel images

arXiv: https://arxiv.org/abs/1911.04432 · HTML: https://mlsanity.com/paper/1911.04432/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Nov 2019.

## Built on by (most cited first, 30 of 41)

- [DTFD-MIL: Double-Tier Feature Distillation Multiple Instance Learning for Histopathology Whole Slide Image Classification](https://mlsanity.com/paper/2203.12081/) (2022)
- [Artificial Intelligence for Digital and Computational Pathology](https://mlsanity.com/paper/2401.06148/) (2023)
- [Attention-Challenging Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2311.07125/) (2023)
- [Multiple Instance Learning Framework with Masked Hard Instance Mining for Whole Slide Image Classification](https://mlsanity.com/paper/2307.15254/) (2023)
- [Hierarchical Graph Representations in Digital Pathology](https://mlsanity.com/paper/2102.11057/) (2021)
- [DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer](https://mlsanity.com/paper/2107.09405/) (2021)
- [Towards Large-Scale Training of Pathology Foundation Models](https://mlsanity.com/paper/2404.15217/) (2024)
- [An Aggregation of Aggregation Methods in Computational Pathology](https://mlsanity.com/paper/2211.01256/) (2022)
- [Hard-Attention for Scalable Image Classification](https://mlsanity.com/paper/2102.10212/) (2021)
- [Efficient Classification of Very Large Images with Tiny Objects](https://mlsanity.com/paper/2106.02694/) (2021)
- [Learning to Holistically Detect Bridges from Large-Size VHR Remote Sensing Imagery](https://mlsanity.com/paper/2312.02481/) (2023)
- [Gigapixel Whole-Slide Images Classification using Locally Supervised Learning](https://mlsanity.com/paper/2207.08267/) (2022)
- [Prompt-MIL: Boosting Multi-Instance Learning Schemes via Task-specific Prompt Tuning](https://mlsanity.com/paper/2303.12214/) (2023)
- [Iterative Patch Selection for High-Resolution Image Recognition](https://mlsanity.com/paper/2210.13007/) (2022)
- [SparseConvMIL: Sparse Convolutional Context-Aware Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2105.02726/) (2021)
- [Multi-View Hypercomplex Learning for Breast Cancer Screening](https://mlsanity.com/paper/2204.05798/) (2022)
- [Generalizing Medical Image Representations via Quaternion Wavelet Networks](https://mlsanity.com/paper/2310.10224/) (2023)
- [SAM-MIL: A Spatial Contextual Aware Multiple Instance Learning Approach for Whole Slide Image Classification](https://mlsanity.com/paper/2407.17689/) (2024)
- [Learning from Data Streams: An Overview and Update](https://mlsanity.com/paper/2212.14720/) (2022)
- [End-to-end Multiple Instance Learning with Gradient Accumulation](https://mlsanity.com/paper/2203.03981/) (2022)
- [Attention De-sparsification Matters: Inducing Diversity in Digital Pathology Representation Learning](https://mlsanity.com/paper/2309.06439/) (2023)
- [TinyChirp: Bird Song Recognition Using TinyML Models on Low-power Wireless Acoustic Sensors](https://mlsanity.com/paper/2407.21453/) (2024)
- [Diagnostic Text-guided Representation Learning in Hierarchical Classification for Pathological Whole Slide Image](https://mlsanity.com/paper/2411.10709/) (2024)
- [Pseudo-Data based Self-Supervised Federated Learning for Classification of Histopathological Images](https://mlsanity.com/paper/2205.15530/) (2022)
- [Local Padding in Patch-Based GANs for Seamless Infinite-Sized Texture Synthesis](https://mlsanity.com/paper/2309.02340/) (2023)
- [Unleashing the Infinity Power of Geometry: A Novel Geometry-Aware Transformer (GOAT) for Whole Slide Histopathology Image Analysis](https://mlsanity.com/paper/2402.05373/) (2024)
- [msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML](https://mlsanity.com/paper/2505.11483/) (2025)
- [Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review](https://mlsanity.com/paper/2211.14847/) (2022)
- [Patch Gradient Descent: Training Neural Networks on Very Large Images](https://mlsanity.com/paper/2301.13817/) (2023)
- [Beyond Multiple Instance Learning: Full Resolution All-In-Memory End-To-End Pathology Slide Modeling](https://mlsanity.com/paper/2403.04865/) (2024)

## Abstract (arXiv)

Due to memory constraints on current hardware, most convolution neural networks (CNN) are trained on sub-megapixel images. For example, most popular datasets in computer vision contain images much less than a megapixel in size (0.09MP for ImageNet and 0.001MP for CIFAR-10). In some domains such as medical imaging, multi-megapixel images are needed to identify the presence of disease accurately. We propose a novel method to directly train convolutional neural networks using any input image size end-to-end. This method exploits the locality of most operations in modern convolutional neural networks by performing the forward and backward pass on smaller tiles of the image. In this work, we show a proof of concept using images of up to 66-megapixels (8192x8192), saving approximately 50GB of memory per image. Using two public challenge datasets, we demonstrate that CNNs can learn to extract relevant information from these large images and benefit from increasing resolution. We improved the area under the receiver-operating characteristic curve from 0.580 (4MP) to 0.706 (66MP) for metastasis detection in breast cancer (CAMELYON17). We also obtained a Spearman correlation metric approaching state-of-the-art performance on the TUPAC16 dataset, from 0.485 (1MP) to 0.570 (16MP). Code to reproduce a subset of the experiments is available at https://github.com/DIAGNijmegen/StreamingCNN.