# The Lie Derivative for Measuring Learned Equivariance

arXiv: https://arxiv.org/abs/2210.02984 · HTML: https://mlsanity.com/paper/2210.02984/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 93 references that are in the map.
- First posted on arXiv 6 Oct 2022.

## Builds on (most cited first, 30 of 74)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Xception: Deep Learning with Depthwise Separable Convolutions](https://mlsanity.com/paper/1610.02357/) (2016)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Alias-Free Generative Adversarial Networks](https://mlsanity.com/paper/2106.12423/) (2021)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [EfficientNetV2: Smaller Models and Faster Training](https://mlsanity.com/paper/2104.00298/) (2021)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [MnasNet: Platform-Aware Neural Architecture Search for Mobile](https://mlsanity.com/paper/1807.11626/) (2018)
- [CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification](https://mlsanity.com/paper/2103.14899/) (2021)
- [Going deeper with Image Transformers](https://mlsanity.com/paper/2103.17239/) (2021)
- [E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials](https://mlsanity.com/paper/2101.03164/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)

## Built on by (most cited first, 30 of 54)

- [Large Language Models Are Zero-Shot Time Series Forecasters](https://mlsanity.com/paper/2310.07820/) (2023)
- [Fine-Tuned Language Models Generate Stable Inorganic Materials as Text](https://mlsanity.com/paper/2402.04379/) (2024)
- [Two for One: Diffusion Models and Force Fields for Coarse-Grained Molecular Dynamics](https://mlsanity.com/paper/2302.00600/) (2023)
- [Orb: A Fast, Scalable Neural Network Potential](https://mlsanity.com/paper/2410.22570/) (2024)
- [The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning](https://mlsanity.com/paper/2304.05366/) (2023)
- [Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learning](https://mlsanity.com/paper/2305.19913/) (2023)
- [Swallowing the Bitter Pill: Simplified Scalable Conformer Generation](https://mlsanity.com/paper/2311.17932/) (2023)
- [Equivariant Adaptation of Large Pretrained Models](https://mlsanity.com/paper/2310.01647/) (2023)
- [3D molecule generation by denoising voxel grids](https://mlsanity.com/paper/2306.07473/) (2023)
- [LieGG: Studying Learned Lie Group Generators](https://mlsanity.com/paper/2210.04345/) (2022)
- [Learning General-Purpose Biomedical Volume Representations using Randomized Synthesis](https://mlsanity.com/paper/2411.02372/) (2024)
- [Improving Equivariant Model Training via Constraint Relaxation](https://mlsanity.com/paper/2408.13242/) (2024)
- [Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principles](https://mlsanity.com/paper/2301.05638/) (2023)
- [Steerers: A framework for rotation equivariant keypoint descriptors](https://mlsanity.com/paper/2312.02152/) (2023)
- [Structure-based drug design by denoising voxel grids](https://mlsanity.com/paper/2405.03961/) (2024)
- [Learning Infinitesimal Generators of Continuous Symmetries from Data](https://mlsanity.com/paper/2410.21853/) (2024)
- [Relaxed Equivariance via Multitask Learning](https://mlsanity.com/paper/2410.17878/) (2024)
- [Probing Equivariance and Symmetry Breaking in Convolutional Networks](https://mlsanity.com/paper/2501.01999/) (2025)
- [Almost Equivariance via Lie Algebra Convolutions](https://mlsanity.com/paper/2310.13164/) (2023)
- [SE(3)-Equivariant and Noise-Invariant 3D Rigid Motion Tracking in Brain MRI](https://mlsanity.com/paper/2312.13534/) (2023)
- [Towards a unified and verified understanding of group-operation networks](https://mlsanity.com/paper/2410.07476/) (2024)
- [On the Ability of Deep Networks to Learn Symmetries from Data: A Neural Kernel Theory](https://mlsanity.com/paper/2412.11521/) (2024)
- [Learning (Approximately) Equivariant Networks via Constrained Optimization](https://mlsanity.com/paper/2505.13631/) (2025)
- [Reviving Shift Equivariance in Vision Transformers](https://mlsanity.com/paper/2306.07470/) (2023)
- [Investigating how ReLU-networks encode symmetries](https://mlsanity.com/paper/2305.17017/) (2023)
- [Flopping for FLOPs: Leveraging equivariance for computational efficiency](https://mlsanity.com/paper/2502.05169/) (2025)
- [The False Promise of Zero-Shot Super-Resolution in Machine-Learned Operators](https://mlsanity.com/paper/2510.06646/) (2025)
- [Identifiable Equivariant Networks are Layerwise Equivariant](https://mlsanity.com/paper/2601.21645/) (2026)
- [Gauge-invariant representation holonomy](https://mlsanity.com/paper/2601.21653/) (2026)
- [Affine steerers for structured keypoint description](https://mlsanity.com/paper/2408.14186/) (2024)

## Abstract (arXiv)

Equivariance guarantees that a model's predictions capture key symmetries in data. When an image is translated or rotated, an equivariant model's representation of that image will translate or rotate accordingly. The success of convolutional neural networks has historically been tied to translation equivariance directly encoded in their architecture. The rising success of vision transformers, which have no explicit architectural bias towards equivariance, challenges this narrative and suggests that augmentations and training data might also play a significant role in their performance. In order to better understand the role of equivariance in recent vision models, we introduce the Lie derivative, a method for measuring equivariance with strong mathematical foundations and minimal hyperparameters. Using the Lie derivative, we study the equivariance properties of hundreds of pretrained models, spanning CNNs, transformers, and Mixer architectures. The scale of our analysis allows us to separate the impact of architecture from other factors like model size or training method. Surprisingly, we find that many violations of equivariance can be linked to spatial aliasing in ubiquitous network layers, such as pointwise non-linearities, and that as models get larger and more accurate they tend to display more equivariance, regardless of architecture. For example, transformers can be more equivariant than convolutional neural networks after training.