# Permutation Equivariance of Transformers and Its Applications

arXiv: https://arxiv.org/abs/2304.07735 · HTML: https://mlsanity.com/paper/2304.07735/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2023.

## Built on by (most cited first, 20 of 20)

- [SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding](https://mlsanity.com/paper/2507.08402/) (2025)
- [SketchDNN: Joint Continuous-Discrete Diffusion for CAD Sketch Generation](https://mlsanity.com/paper/2507.11579/) (2025)
- [Bi-Orthogonal Factor Decomposition for Vision Transformers](https://mlsanity.com/paper/2601.05328/) (2026)
- [REOrdering Patches Improves Vision Models](https://mlsanity.com/paper/2505.23751/) (2025)
- [Privacy-Preserving Split Learning with Vision Transformers using Patch-Wise Random and Noisy CutMix](https://mlsanity.com/paper/2408.01040/) (2024)
- [An analytic theory of convolutional neural network inverse problems solvers](https://mlsanity.com/paper/2601.10334/) (2026)
- [TokenMark: A Modality-Agnostic Watermark for Pre-trained Transformers](https://mlsanity.com/paper/2403.05842/) (2024)
- [MIST: Mutual Information Estimation Via Supervised Training](https://mlsanity.com/paper/2511.18945/) (2025)
- [Names Don't Matter: Symbol-Invariant Transformer for Open-Vocabulary Learning](https://mlsanity.com/paper/2601.23169/) (2026)
- [TopoFlow: Topography-aware Pollutant Flow Learning for High-Resolution Air Quality Prediction](https://mlsanity.com/paper/2602.16821/) (2026)
- [Weights Shuffling for Improving DPSGD in Transformer-based Models](https://mlsanity.com/paper/2407.15414/) (2024)
- [Enhancing Privacy in ControlNet and Stable Diffusion via Split Learning](https://mlsanity.com/paper/2409.08503/) (2024)
- [Physical foundations for trustworthy medical imaging: a review for artificial intelligence researchers](https://mlsanity.com/paper/2505.02843/) (2025)
- [What do language models model? Transformers, automata, and the format of thought](https://mlsanity.com/paper/2508.18598/) (2025)
- [AI/ML based Joint Source and Channel Coding for HARQ-ACK Payload](https://mlsanity.com/paper/2511.19943/) (2025)
- [Symmetry Reveals Layerwise Dynamics: How Transformers Perform In-Context Classification](https://mlsanity.com/paper/2604.11613/) (2026)
- [LLM Compression with Jointly Optimizing Architectural and Quantization choices](https://mlsanity.com/paper/2606.04063/) (2026)
- [Geometric 2D Scene Graph Generation](https://mlsanity.com/paper/2607.22325/) (2026)
- [WithEveryone: Unified Planning and Identity Grounding for Group Image Generation](https://mlsanity.com/paper/2608.20336/) (2026)
- [Position Matters: Feature Inversion Attacks in ViT Split Inference with Token Reduction and Shuffling](https://mlsanity.com/paper/2609.01232/) (2026)

## Abstract (arXiv)

Revolutionizing the field of deep learning, Transformer-based models have achieved remarkable performance in many tasks. Recent research has recognized these models are robust to shuffling but are limited to inter-token permutation in the forward propagation. In this work, we propose our definition of permutation equivariance, a broader concept covering both inter- and intra- token permutation in the forward and backward propagation of neural networks. We rigorously proved that such permutation equivariance property can be satisfied on most vanilla Transformer-based models with almost no adaptation. We examine the property over a range of state-of-the-art models including ViT, Bert, GPT, and others, with experimental validations. Further, as a proof-of-concept, we explore how real-world applications including privacy-enhancing split learning, and model authorization, could exploit the permutation equivariance property, which implicates wider, intriguing application scenarios.