# Manitest: Are classifiers really invariant?

arXiv: https://arxiv.org/abs/1507.06535 · HTML: https://mlsanity.com/paper/1507.06535/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2015.

## Built on by (most cited first, 27 of 27)

- [Editing a classifier by rewriting its prediction rules](https://mlsanity.com/paper/2112.01008/) (2021)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Just Rotate it: Deploying Backdoor Attacks via Rotation Transformation](https://mlsanity.com/paper/2207.10825/) (2022)
- [Attacks in Adversarial Machine Learning: A Systematic Survey from the Life-cycle Perspective](https://mlsanity.com/paper/2302.09457/) (2023)
- [A Survey of Robustness and Safety of 2D and 3D Deep Learning Models Against Adversarial Attacks](https://mlsanity.com/paper/2310.00633/) (2023)
- [CausalAgents: A Robustness Benchmark for Motion Forecasting using Causal Relationships](https://mlsanity.com/paper/2207.03586/) (2022)
- [Scale-invariant scale-channel networks: Deep networks that generalise to previously unseen scales](https://mlsanity.com/paper/2106.06418/) (2021)
- [Adversarial Examples on Segmentation Models Can be Easy to Transfer](https://mlsanity.com/paper/2111.11368/) (2021)
- [Relative stability toward diffeomorphisms indicates performance in deep nets](https://mlsanity.com/paper/2105.02468/) (2021)
- [Anti-aliasing Deep Image Classifiers using Novel Depth Adaptive Blurring and Activation Function](https://mlsanity.com/paper/2110.00899/) (2021)
- [Measuring Representational Robustness of Neural Networks Through Shared Invariances](https://mlsanity.com/paper/2206.11939/) (2022)
- [AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-directional Metric Learning](https://mlsanity.com/paper/2108.06017/) (2021)
- [Adversarially Robust Learning with Tolerance](https://mlsanity.com/paper/2203.00849/) (2022)
- [Contemplating real-world object classification](https://mlsanity.com/paper/2103.05137/) (2021)
- [Generating Unrestricted Adversarial Examples via Three Parameters](https://mlsanity.com/paper/2103.07640/) (2021)
- [Certifying Model Accuracy under Distribution Shifts](https://mlsanity.com/paper/2201.12440/) (2022)
- [How deep convolutional neural networks lose spatial information with training](https://mlsanity.com/paper/2210.01506/) (2022)
- [Exploring Robust Features for Improving Adversarial Robustness](https://mlsanity.com/paper/2309.04650/) (2023)
- [Generalization to translation shifts: a study in architectures and augmentations](https://mlsanity.com/paper/2207.02349/) (2022)
- [Natural Perturbed Training for General Robustness of Neural Network Classifiers](https://mlsanity.com/paper/2103.11372/) (2021)
- [SHIFT15M: Fashion-specific dataset for set-to-set matching with several distribution shifts](https://mlsanity.com/paper/2108.12992/) (2021)
- [Resource-Efficient Invariant Networks: Exponential Gains by Unrolled Optimization](https://mlsanity.com/paper/2203.05006/) (2022)
- [An Analysis of Causal Effect Estimation using Outcome Invariant Data Augmentation](https://mlsanity.com/paper/2510.25128/) (2025)
- [Verifying Attention Robustness of Deep Neural Networks against Semantic Perturbations](https://mlsanity.com/paper/2207.05902/) (2022)
- [Invariance Measures for Neural Networks](https://mlsanity.com/paper/2310.17404/) (2023)
- [Wiggling Weights to Improve the Robustness of Classifiers](https://mlsanity.com/paper/2111.09779/) (2021)
- [Potential Auto-driving Threat: Universal Rain-removal Attack](https://mlsanity.com/paper/2211.09959/) (2022)

## Abstract (arXiv)

Invariance to geometric transformations is a highly desirable property of automatic classifiers in many image recognition tasks. Nevertheless, it is unclear to which extent state-of-the-art classifiers are invariant to basic transformations such as rotations and translations. This is mainly due to the lack of general methods that properly measure such an invariance. In this paper, we propose a rigorous and systematic approach for quantifying the invariance to geometric transformations of any classifier. Our key idea is to cast the problem of assessing a classifier's invariance as the computation of geodesics along the manifold of transformed images. We propose the Manitest method, built on the efficient Fast Marching algorithm to compute the invariance of classifiers. Our new method quantifies in particular the importance of data augmentation for learning invariance from data, and the increased invariance of convolutional neural networks with depth. We foresee that the proposed generic tool for measuring invariance to a large class of geometric transformations and arbitrary classifiers will have many applications for evaluating and comparing classifiers based on their invariance, and help improving the invariance of existing classifiers.