# DeepPermNet: Visual Permutation Learning

arXiv: https://arxiv.org/abs/1704.02729 · HTML: https://mlsanity.com/paper/1704.02729/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2017.

## Built on by (most cited first, 30 of 32)

- [Self-Supervised Learning for Videos: A Survey](https://mlsanity.com/paper/2207.00419/) (2022)
- [Tutorial on amortized optimization](https://mlsanity.com/paper/2202.00665/) (2022)
- [A Generalist Neural Algorithmic Learner](https://mlsanity.com/paper/2209.11142/) (2022)
- [Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2510.27606/) (2025)
- [Geographical Knowledge-driven Representation Learning for Remote Sensing Images](https://mlsanity.com/paper/2107.05276/) (2021)
- [A Unified Framework for Implicit Sinkhorn Differentiation](https://mlsanity.com/paper/2205.06688/) (2022)
- [Jigsaw-ViT: Learning Jigsaw Puzzles in Vision Transformer](https://mlsanity.com/paper/2207.11971/) (2022)
- [Location-Aware Self-Supervised Transformers for Semantic Segmentation](https://mlsanity.com/paper/2212.02400/) (2022)
- [DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions](https://mlsanity.com/paper/2309.03576/) (2023)
- [SPANet: Generalized Permutationless Set Assignment for Particle Physics using Symmetry Preserving Attention](https://mlsanity.com/paper/2106.03898/) (2021)
- [JigsawGAN: Auxiliary Learning for Solving Jigsaw Puzzles with Generative Adversarial Networks](https://mlsanity.com/paper/2101.07555/) (2021)
- [LinSATNet: The Positive Linear Satisfiability Neural Networks](https://mlsanity.com/paper/2407.13917/) (2024)
- [Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking](https://mlsanity.com/paper/2103.09879/) (2021)
- [Robust Partial-to-Partial Point Cloud Registration in a Full Range](https://mlsanity.com/paper/2111.15606/) (2021)
- [DiffPrep: Differentiable Data Preprocessing Pipeline Search for Learning over Tabular Data](https://mlsanity.com/paper/2308.10915/) (2023)
- [Contrastive Learning for Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2105.03117/) (2021)
- [Boosting Generative Zero-Shot Learning by Synthesizing Diverse Features with Attribute Augmentation](https://mlsanity.com/paper/2112.12573/) (2021)
- [GLinSAT: The General Linear Satisfiability Neural Network Layer By Accelerated Gradient Descent](https://mlsanity.com/paper/2409.17500/) (2024)
- [Manual-PA: Learning 3D Part Assembly from Instruction Diagrams](https://mlsanity.com/paper/2411.18011/) (2024)
- [Contextual Text Block Detection towards Scene Text Understanding](https://mlsanity.com/paper/2207.12955/) (2022)
- [SymmetricDiffusers: Learning Discrete Diffusion on Finite Symmetric Groups](https://mlsanity.com/paper/2410.02942/) (2024)
- [Unlabeled Principal Component Analysis and Matrix Completion](https://mlsanity.com/paper/2101.09446/) (2021)
- [Kissing to Find a Match: Efficient Low-Rank Permutation Representation](https://mlsanity.com/paper/2308.13252/) (2023)
- [Semi-Supervised Domain Adaptation via Adaptive and Progressive Feature Alignment](https://mlsanity.com/paper/2106.02845/) (2021)
- [Low-rank Matrix Recovery With Unknown Correspondence](https://mlsanity.com/paper/2110.07959/) (2021)
- [DISGO: Automatic End-to-End Evaluation for Scene Text OCR](https://mlsanity.com/paper/2308.13173/) (2023)
- [Differentiable Random Partition Models](https://mlsanity.com/paper/2305.16841/) (2023)
- [Set Interdependence Transformer: Set-to-Sequence Neural Networks for Permutation Learning and Structure Prediction](https://mlsanity.com/paper/2206.03720/) (2022)
- [Self-Ordering Point Clouds](https://mlsanity.com/paper/2304.00961/) (2023)
- [Zero-Shot Learning with Subsequence Reordering Pretraining for Compound-Protein Interaction](https://mlsanity.com/paper/2507.20925/) (2025)

## Abstract (arXiv)

We present a principled approach to uncover the structure of visual data by solving a novel deep learning task coined visual permutation learning. The goal of this task is to find the permutation that recovers the structure of data from shuffled versions of it. In the case of natural images, this task boils down to recovering the original image from patches shuffled by an unknown permutation matrix. Unfortunately, permutation matrices are discrete, thereby posing difficulties for gradient-based methods. To this end, we resort to a continuous approximation of these matrices using doubly-stochastic matrices which we generate from standard CNN predictions using Sinkhorn iterations. Unrolling these iterations in a Sinkhorn network layer, we propose DeepPermNet, an end-to-end CNN model for this task. The utility of DeepPermNet is demonstrated on two challenging computer vision problems, namely, (i) relative attributes learning and (ii) self-supervised representation learning. Our results show state-of-the-art performance on the Public Figures and OSR benchmarks for (i) and on the classification and segmentation tasks on the PASCAL VOC dataset for (ii).