# Linearized Multi-Sampling for Differentiable Image Transformation

arXiv: https://arxiv.org/abs/1901.07124 · HTML: https://mlsanity.com/paper/1901.07124/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Jan 2019.

## Built on by (most cited first, 11 of 11)

- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [nerf2nerf: Pairwise Registration of Neural Radiance Fields](https://mlsanity.com/paper/2211.01600/) (2022)
- [Context-aware Attentional Pooling (CAP) for Fine-grained Visual Classification](https://mlsanity.com/paper/2101.06635/) (2021)
- [PointMBF: A Multi-scale Bidirectional Fusion Network for Unsupervised RGB-D Point Cloud Registration](https://mlsanity.com/paper/2308.04782/) (2023)
- [VTAE: Variational Transformer Autoencoder with Manifolds Learning](https://mlsanity.com/paper/2304.00948/) (2023)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)
- [Human Pose Transfer with Augmented Disentangled Feature Consistency](https://mlsanity.com/paper/2107.10984/) (2021)
- [Magnifying Networks for Images with Billions of Pixels](https://mlsanity.com/paper/2112.06121/) (2021)
- [Improved monocular depth prediction using distance transform over pre-semantic contours with self-supervised neural networks](https://mlsanity.com/paper/2605.08320/) (2026)
- [HashEncoding: Autoencoding with Multiscale Coordinate Hashing](https://mlsanity.com/paper/2211.15894/) (2022)
- [Entropy Transformer Networks: A Learning Approach via Tangent Bundle Data Manifold](https://mlsanity.com/paper/2307.12517/) (2023)

## Abstract (arXiv)

We propose a novel image sampling method for differentiable image transformation in deep neural networks. The sampling schemes currently used in deep learning, such as Spatial Transformer Networks, rely on bilinear interpolation, which performs poorly under severe scale changes, and more importantly, results in poor gradient propagation. This is due to their strict reliance on direct neighbors. Instead, we propose to generate random auxiliary samples in the vicinity of each pixel in the sampled image, and create a linear approximation with their intensity values. We then use this approximation as a differentiable formula for the transformed image. We demonstrate that our approach produces more representative gradients with a wider basin of convergence for image alignment, which leads to considerable performance improvements when training networks for classification tasks. This is not only true under large downsampling, but also when there are no scale changes. We compare our approach with multi-scale sampling and show that we outperform it. We then demonstrate that our improvements to the sampler are compatible with other tangential improvements to Spatial Transformer Networks and that it further improves their performance.