# How Shift Equivariance Impacts Metric Learning for Instance Segmentation

arXiv: https://arxiv.org/abs/2101.05846 · HTML: https://mlsanity.com/paper/2101.05846/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 14 Jan 2021.

## Builds on (most cited first, 18 of 18)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Why do deep convolutional networks generalize so poorly to small image transformations?](https://mlsanity.com/paper/1805.12177/) (2018)
- [Cell Detection with Star-convex Polygons](https://mlsanity.com/paper/1806.03535/) (2018)
- [On Translation Invariance in CNNs: Convolutional Layers can Exploit Absolute Spatial Location](https://mlsanity.com/paper/2003.07064/) (2020)
- [Semantic Instance Segmentation with a Discriminative Loss Function](https://mlsanity.com/paper/1708.02551/) (2017)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)
- [Superhuman Accuracy on the SNEMI3D Connectomics Challenge](https://mlsanity.com/paper/1706.00120/) (2017)
- [Mind the Pad -- CNNs can Develop Blind Spots](https://mlsanity.com/paper/2010.02178/) (2020)
- [Tiling and Stitching Segmentation Output for Remote Sensing: Basic Challenges and Recommendations](https://mlsanity.com/paper/1805.12219/) (2018)
- [Semi-convolutional Operators for Instance Segmentation](https://mlsanity.com/paper/1807.10712/) (2018)
- [Instance Segmentation of Biological Images Using Harmonic Embeddings](https://mlsanity.com/paper/1904.05257/) (2019)
- [Instance Segmentation of Biomedical Images with an Object-aware Embedding Learned with Local Constraints](https://mlsanity.com/paper/2004.09821/) (2020)
- [Probabilistic Deep Learning for Instance Segmentation](https://mlsanity.com/paper/2008.10678/) (2020)
- [An Auxiliary Task for Learning Nuclei Segmentation in 3D Microscopy Images](https://mlsanity.com/paper/2002.02857/) (2020)
- [Instance Segmentation by Deep Coloring](https://mlsanity.com/paper/1807.10007/) (2018)

## Built on by (most cited first, 7 of 7)

- [Learning Image Priors through Patch-based Diffusion Models for Solving Inverse Problems](https://mlsanity.com/paper/2406.02462/) (2024)
- [Panoptic segmentation with highly imbalanced semantic labels](https://mlsanity.com/paper/2203.11692/) (2022)
- [Tiling artifacts and trade-offs of feature normalization in the segmentation of large biological images](https://mlsanity.com/paper/2503.19545/) (2025)
- [ShapeEmbed: a self-supervised learning framework for 2D contour quantification](https://mlsanity.com/paper/2507.01009/) (2025)
- [Dual-level Interaction for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2307.07972/) (2023)
- [Stitching and dimensionality effects on large artificially generated volume datasets](https://mlsanity.com/paper/2606.20095/) (2026)
- [SWITi: Quantifying and Reducing Tiling Artifacts with Sliding Window Inner Tiling](https://mlsanity.com/paper/2607.18990/) (2026)

## Abstract (arXiv)

Metric learning has received conflicting assessments concerning its suitability for solving instance segmentation tasks. It has been dismissed as theoretically flawed due to the shift equivariance of the employed CNNs and their respective inability to distinguish same-looking objects. Yet it has been shown to yield state of the art results for a variety of tasks, and practical issues have mainly been reported in the context of tile-and-stitch approaches, where discontinuities at tile boundaries have been observed. To date, neither of the reported issues have undergone thorough formal analysis. In our work, we contribute a comprehensive formal analysis of the shift equivariance properties of encoder-decoder-style CNNs, which yields a clear picture of what can and cannot be achieved with metric learning in the face of same-looking objects. In particular, we prove that a standard encoder-decoder network that takes d-dimensional images as input, with l pooling layers and pooling factor f, has the capacity to distinguish at most fdl same-looking objects, and we show that this upper limit can be reached. Furthermore, we show that to avoid discontinuities in a tile-and-stitch approach, assuming standard batch size 1, it is necessary to employ valid convolutions in combination with a training output window size strictly greater than fl, while at test-time it is necessary to crop tiles to size n· fl before stitching, with n≥ 1. We complement these theoretical findings by discussing a number of insightful special cases for which we show empirical results on synthetic data.