# Entropy Transformer Networks: A Learning Approach via Tangent Bundle Data Manifold

arXiv: https://arxiv.org/abs/2307.12517 · HTML: https://mlsanity.com/paper/2307.12517/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 24 Jul 2023.

## Builds on (most cited first, 24 of 24)

- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Group Equivariant Convolutional Networks](https://mlsanity.com/paper/1602.07576/) (2016)
- [E(n) Equivariant Graph Neural Networks](https://mlsanity.com/paper/2102.09844/) (2021)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Harmonic Networks: Deep Translation and Rotation Equivariance](https://mlsanity.com/paper/1612.04642/) (2016)
- [Gauge Equivariant Convolutional Networks and the Icosahedral CNN](https://mlsanity.com/paper/1902.04615/) (2019)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [Single-Shot Refinement Neural Network for Object Detection](https://mlsanity.com/paper/1711.06897/) (2017)
- [Towards the first adversarially robust neural network model on MNIST](https://mlsanity.com/paper/1805.09190/) (2018)
- [Approximately Equivariant Networks for Imperfectly Symmetric Dynamics](https://mlsanity.com/paper/2201.11969/) (2022)
- [Inverse Compositional Spatial Transformer Networks](https://mlsanity.com/paper/1612.03897/) (2016)
- [Warped Convolutions: Efficient Invariance to Spatial Transformations](https://mlsanity.com/paper/1609.04382/) (2016)
- [Equivariance versus Augmentation for Spherical Images](https://mlsanity.com/paper/2202.03990/) (2022)
- [Scaling-Translation-Equivariant Networks with Decomposed Convolutional Filters](https://mlsanity.com/paper/1909.11193/) (2019)
- [Equivariant Hypergraph Neural Networks](https://mlsanity.com/paper/2208.10428/) (2022)
- [Linearized Multi-Sampling for Differentiable Image Transformation](https://mlsanity.com/paper/1901.07124/) (2019)
- [Learning Conditional Variational Autoencoders with Missing Covariates](https://mlsanity.com/paper/2203.01218/) (2022)
- [VTAE: Variational Transformer Autoencoder with Manifolds Learning](https://mlsanity.com/paper/2304.00948/) (2023)
- [Learning Equality Constraints for Motion Planning on Manifolds](https://mlsanity.com/paper/2009.11852/) (2020)
- [Moving Frame Net: SE(3)-Equivariant Network for Volumes](https://mlsanity.com/paper/2211.03420/) (2022)
- [Zero-Shot Learning by Generating Pseudo Feature Representations](https://mlsanity.com/paper/1703.06389/) (2017)
- [Image Projective Invariants](https://mlsanity.com/paper/1707.05950/) (2017)
- [Learning Invariant Representations for Equivariant Neural Networks Using Orthogonal Moments](https://mlsanity.com/paper/2209.10944/) (2022)

## Abstract (arXiv)

This paper focuses on an accurate and fast interpolation approach for image transformation employed in the design of CNN architectures. Standard Spatial Transformer Networks (STNs) use bilinear or linear interpolation as their interpolation, with unrealistic assumptions about the underlying data distributions, which leads to poor performance under scale variations. Moreover, STNs do not preserve the norm of gradients in propagation due to their dependency on sparse neighboring pixels. To address this problem, a novel Entropy STN (ESTN) is proposed that interpolates on the data manifold distributions. In particular, random samples are generated for each pixel in association with the tangent space of the data manifold and construct a linear approximation of their intensity values with an entropy regularizer to compute the transformer parameters. A simple yet effective technique is also proposed to normalize the non-zero values of the convolution operation, to fine-tune the layers for gradients' norm-regularization during training. Experiments on challenging benchmarks show that the proposed ESTN can improve predictive accuracy over a range of computer vision tasks, including image reconstruction, and classification, while reducing the computational cost.