# REOrdering Patches Improves Vision Models

arXiv: https://arxiv.org/abs/2505.23751 · HTML: https://mlsanity.com/paper/2505.23751/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 14 of 14)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Functional Map of the World](https://mlsanity.com/paper/1711.07846/) (2017)
- [Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacement](https://mlsanity.com/paper/1903.06059/) (2019)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation](https://mlsanity.com/paper/2110.07858/) (2021)
- [Autoregressive Pretraining with Mamba in Vision](https://mlsanity.com/paper/2406.07537/) (2024)
- [Permutation Equivariance of Transformers and Its Applications](https://mlsanity.com/paper/2304.07735/) (2023)
- [xT: Nested Tokenization for Larger Context in Large Images](https://mlsanity.com/paper/2403.01915/) (2024)
- [Patch Ranking: Efficient CLIP by Learning to Rank Local Patches](https://mlsanity.com/paper/2409.14607/) (2024)

## Built on by (most cited first, 3 of 3)

- [Neighbor-Aware Token Reduction via Hilbert Curve for Vision Transformers](https://mlsanity.com/paper/2512.22760/) (2025)
- [Progressive Checkerboards for Autoregressive Multiscale Image Generation](https://mlsanity.com/paper/2602.03811/) (2026)
- [AnaDiffusion: Anatomically CompositionalLatent Diffusion for Controllable 3D Brain MRI Generation](https://mlsanity.com/paper/2608.23014/) (2026)

## Abstract (arXiv)

Sequence models such as transformers require inputs to be represented as one-dimensional sequences. In vision, this typically involves flattening images using a fixed row-major (raster-scan) order. While full self-attention is permutation-equivariant, modern long-sequence transformers increasingly rely on architectural approximations that break this invariance and introduce sensitivity to patch ordering. We show that patch order significantly affects model performance in such settings, with simple alternatives like column-major or Hilbert curves yielding notable accuracy shifts. Motivated by this, we propose REOrder, a two-stage framework for discovering task-optimal patch orderings. First, we derive an information-theoretic prior by evaluating the compressibility of various patch sequences. Then, we learn a policy over permutations by optimizing a Plackett-Luce policy using REINFORCE. This approach enables efficient learning in a combinatorial permutation space. REOrder improves top-1 accuracy over row-major ordering on ImageNet-1K by up to 3.01% and Functional Map of the World by 13.35%.