# Multiscale Training of Convolutional Neural Networks

arXiv: https://arxiv.org/abs/2501.12739 · HTML: https://mlsanity.com/paper/2501.12739/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 22 Jan 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Neural Operator: Graph Kernel Network for Partial Differential Equations](https://mlsanity.com/paper/2003.03485/) (2020)
- [Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning](https://mlsanity.com/paper/1611.01929/) (2016)
- [Variance Reduction in SGD by Distributed Importance Sampling](https://mlsanity.com/paper/1511.06481/) (2015)
- [MgNet: A Unified Framework of Multigrid and Convolutional Neural Network](https://mlsanity.com/paper/1901.10415/) (2019)
- [Wavelet Convolutional Neural Networks](https://mlsanity.com/paper/1805.08620/) (2018)
- [Wavelet Feature Maps Compression for Image-to-Image CNNs](https://mlsanity.com/paper/2205.12268/) (2022)
- [Deep Multiscale Model Learning](https://mlsanity.com/paper/1806.04830/) (2018)
- [MGIC: Multigrid-in-Channels Neural Network Architectures](https://mlsanity.com/paper/2011.09128/) (2020)
- [Learning across scales - A multiscale method for Convolution Neural Networks](https://mlsanity.com/paper/1703.02009/) (2017)
- [Haar Wavelet Feature Compression for Quantized Graph Convolutional Networks](https://mlsanity.com/paper/2110.04824/) (2021)
- [Multilevel Artificial Neural Network Training for Spatially Correlated Learning](https://mlsanity.com/paper/1806.05703/) (2018)
- [Robust Optimization of PDEs with Random Coefficients Using a Multilevel Monte Carlo Method](https://mlsanity.com/paper/1711.02574/) (2017)

## Built on by (most cited first, 3 of 3)

- [Enabling Automatic Differentiation with Mollified Graph Neural Operators](https://mlsanity.com/paper/2504.08277/) (2025)
- [Towards Efficient Training of Graph Neural Networks: A Multiscale Approach](https://mlsanity.com/paper/2503.19666/) (2025)
- [Multi-Level Monte Carlo Training of Neural Operators](https://mlsanity.com/paper/2505.12940/) (2025)

## Abstract (arXiv)

Training convolutional neural networks (CNNs) on high-resolution images is often bottlenecked by the cost of evaluating gradients of the loss on the finest spatial mesh. To address this, we propose Multiscale Gradient Estimation (MGE), a Multilevel Monte Carlo-inspired estimator that expresses the expected gradient on the finest mesh as a telescopic sum of gradients computed on progressively coarser meshes. By assigning larger batches to the cheaper coarse levels, MGE achieves the same variance as single-scale stochastic gradient estimation while reducing the number of fine mesh convolutions by a factor of 4 with each downsampling. We further embed MGE within a Full-Multiscale training algorithm that solves the learning problem on coarse meshes first and "hot-starts" the next finer level, cutting the required fine mesh iterations by an additional order of magnitude. Extensive experiments on image denoising, deblurring, inpainting and super-resolution tasks using UNet, ResNet and ESPCN backbones confirm the practical benefits: Full-Multiscale reduces the computation costs by 4-16x with no significant loss in performance. Together, MGE and Full-Multiscale offer a principled, architecture-agnostic route to accelerate CNN training on high-resolution data without sacrificing accuracy, and they can be combined with other variance-reduction or learning-rate schedules to further enhance scalability.