# Differentiable Sorting Networks for Scalable Sorting and Ranking Supervision

arXiv: https://arxiv.org/abs/2105.04019 · HTML: https://mlsanity.com/paper/2105.04019/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 9 May 2021.

## Builds on (most cited first, 10 of 10)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Order Matters: Sequence to sequence for sets](https://mlsanity.com/paper/1511.06391/) (2015)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [Learning Latent Permutations with Gumbel-Sinkhorn Networks](https://mlsanity.com/paper/1802.08665/) (2018)
- [Differentiable Ranks and Sorting using Optimal Transport](https://mlsanity.com/paper/1905.11885/) (2019)
- [Stochastic Optimization of Sorting Networks via Continuous Relaxations](https://mlsanity.com/paper/1903.08850/) (2019)
- [Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks](https://mlsanity.com/paper/1312.6082/) (2013)

## Built on by (most cited first, 30 of 31)

- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Deep Differentiable Logic Gate Networks](https://mlsanity.com/paper/2210.08277/) (2022)
- [Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities](https://mlsanity.com/paper/2111.08851/) (2021)
- [Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective](https://mlsanity.com/paper/2302.01425/) (2023)
- [Monotonic Differentiable Sorting Networks](https://mlsanity.com/paper/2203.09630/) (2022)
- [Differentiable Top-k Classification Learning](https://mlsanity.com/paper/2206.07290/) (2022)
- [Near, far: Patch-ordering enhances vision foundation models' scene understanding](https://mlsanity.com/paper/2408.11054/) (2024)
- [Learning by Sorting: Self-supervised Learning with Group Ordering Constraints](https://mlsanity.com/paper/2301.02009/) (2023)
- [Data-Efficient Learning with Neural Programs](https://mlsanity.com/paper/2406.06246/) (2024)
- [Beam Tree Recursive Cells](https://mlsanity.com/paper/2305.19999/) (2023)
- [Parallel Algorithms Align with Neural Execution](https://mlsanity.com/paper/2307.04049/) (2023)
- [Adaptive Neural Ranking Framework: Toward Maximized Business Goal for Cascade Ranking Systems](https://mlsanity.com/paper/2310.10462/) (2023)
- [SymmetricDiffusers: Learning Discrete Diffusion on Finite Symmetric Groups](https://mlsanity.com/paper/2410.02942/) (2024)
- [Made to Order: Discovering monotonic temporal changes via self-supervised video ordering](https://mlsanity.com/paper/2404.16828/) (2024)
- [Generalized Neural Sorting Networks with Error-Free Differentiable Swap Functions](https://mlsanity.com/paper/2310.07174/) (2023)
- [Generalizing Stochastic Smoothing for Differentiation and Gradient Estimation](https://mlsanity.com/paper/2410.08125/) (2024)
- [Relational Surrogate Loss Learning](https://mlsanity.com/paper/2202.13197/) (2022)
- [Holistic Utility Preference Learning for Listwise Alignment](https://mlsanity.com/paper/2410.18127/) (2024)
- [Newton Losses: Using Curvature Information for Learning with Differentiable Algorithms](https://mlsanity.com/paper/2410.19055/) (2024)
- [Learning Cascade Ranking as One Network](https://mlsanity.com/paper/2503.09492/) (2025)
- [SoftJAX & SoftTorch: Empowering Automatic Differentiation Libraries with Informative Gradients](https://mlsanity.com/paper/2603.08824/) (2026)
- [Diffsurv: Differentiable sorting for censored time-to-event data](https://mlsanity.com/paper/2304.13594/) (2023)
- [SAMSA: Efficient Transformer for Many Data Modalities](https://mlsanity.com/paper/2408.05391/) (2024)
- [RoCP-GNN: Robust Conformal Prediction for Graph Neural Networks in Node-Classification](https://mlsanity.com/paper/2408.13825/) (2024)
- [On the algorithmic construction of deep ReLU networks](https://mlsanity.com/paper/2506.19104/) (2025)
- [Conformal Correction for Efficiency May be at Odds with Entropy](https://mlsanity.com/paper/2512.02704/) (2025)
- [Structural Incompatibility of Differentiable Sorting and Within-Vector Rank Normalization](https://mlsanity.com/paper/2512.22587/) (2025)
- [Logic-Gated Time-Shared Feedforward Networks for Alternating Finite Automata: Exact Simulation and Learnability](https://mlsanity.com/paper/2604.01228/) (2026)
- [LaPrune: Controllable Differentiable Sparsity at Million Scale](https://mlsanity.com/paper/2608.04057/) (2026)
- [Fast LapSum: Exact Differentiable Top-k at Million Scale](https://mlsanity.com/paper/2608.06912/) (2026)

## Abstract (arXiv)

Sorting and ranking supervision is a method for training neural networks end-to-end based on ordering constraints. That is, the ground truth order of sets of samples is known, while their absolute values remain unsupervised. For that, we propose differentiable sorting networks by relaxing their pairwise conditional swap operations. To address the problems of vanishing gradients and extensive blurring that arise with larger numbers of layers, we propose mapping activations to regions with moderate gradients. We consider odd-even as well as bitonic sorting networks, which outperform existing relaxations of the sorting operation. We show that bitonic sorting networks can achieve stable training on large input sets of up to 1024 elements.