# LapSum -- One Method to Differentiate Them All: Ranking, Sorting and Top-k Selection

arXiv: https://arxiv.org/abs/2503.06242 · HTML: https://mlsanity.com/paper/2503.06242/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Learning A Sparse Transformer Network for Effective Image Deraining](https://mlsanity.com/paper/2303.11950/) (2023)
- [Fast Differentiable Sorting and Ranking](https://mlsanity.com/paper/2002.08871/) (2020)
- [Top-K Off-Policy Correction for a REINFORCE Recommender System](https://mlsanity.com/paper/1812.02353/) (2018)
- [Differentiable Ranks and Sorting using Optimal Transport](https://mlsanity.com/paper/1905.11885/) (2019)
- [Stochastic Optimization of Sorting Networks via Continuous Relaxations](https://mlsanity.com/paper/1903.08850/) (2019)
- [SoftSort: A Continuous Relaxation for the argsort Operator](https://mlsanity.com/paper/2006.16038/) (2020)
- [Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection](https://mlsanity.com/paper/1912.11637/) (2019)
- [Smooth Loss Functions for Deep Top-k Classification](https://mlsanity.com/paper/1802.07595/) (2018)
- [Learning with Noisy Labels via Sparse Regularization](https://mlsanity.com/paper/2108.00192/) (2021)
- [Loss Functions for Top-k Error: Analysis and Insights](https://mlsanity.com/paper/1512.00486/) (2015)
- [Fast, Differentiable and Sparse Top-k: a Convex Analysis Perspective](https://mlsanity.com/paper/2302.01425/) (2023)
- [Monotonic Differentiable Sorting Networks](https://mlsanity.com/paper/2203.09630/) (2022)
- [Cardinality-Aware Set Prediction and Top-k Classification](https://mlsanity.com/paper/2407.07140/) (2024)
- [Differentiable Top-k Classification Learning](https://mlsanity.com/paper/2206.07290/) (2022)
- [Differentiable Top-k Operator with Optimal Transport](https://mlsanity.com/paper/2002.06504/) (2020)
- [Learning by Sorting: Self-supervised Learning with Group Ordering Constraints](https://mlsanity.com/paper/2301.02009/) (2023)
- [Stochastic smoothing of the top-K calibrated hinge loss for deep imbalanced classification](https://mlsanity.com/paper/2202.02193/) (2022)
- [Top-k Classification and Cardinality-Aware Prediction](https://mlsanity.com/paper/2403.19625/) (2024)

## Built on by (most cited first, 13 of 13)

- [VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs](https://mlsanity.com/paper/2510.16598/) (2025)
- [Efficient Transferable Optimal Transport via Min-Sliced Transport Plans](https://mlsanity.com/paper/2511.19741/) (2025)
- [SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs](https://mlsanity.com/paper/2606.17952/) (2026)
- [Differentiable Fast Top-K Selection for Large-Scale Recommendation](https://mlsanity.com/paper/2510.11472/) (2025)
- [SoftSAE: Dynamic Top-K Selection for Adaptive Sparse Autoencoders](https://mlsanity.com/paper/2605.06610/) (2026)
- [DiffPrune: differentiable information throttling for token pruning in vision-language models](https://mlsanity.com/paper/2608.01985/) (2026)
- [Structural Incompatibility of Differentiable Sorting and Within-Vector Rank Normalization](https://mlsanity.com/paper/2512.22587/) (2025)
- [Min Generalized Sliced Gromov Wasserstein: A Scalable Path to Gromov Wasserstein](https://mlsanity.com/paper/2605.13753/) (2026)
- [Interpretability-Guided Soft Pruning of Attention Heads in Vision Transformers](https://mlsanity.com/paper/2608.00264/) (2026)
- [LaPrune: Controllable Differentiable Sparsity at Million Scale](https://mlsanity.com/paper/2608.04057/) (2026)
- [Fast LapSum: Exact Differentiable Top-k at Million Scale](https://mlsanity.com/paper/2608.06912/) (2026)
- [ClasSAE: Class-Aligned Sparse Autoencoders via Differentiable Feature-Class Affinity](https://mlsanity.com/paper/2610.04020/) (2026)
- [Principled Top-k Selection for Language Models with Hybrid Gradients](https://mlsanity.com/paper/2610.04162/) (2026)

## Abstract (arXiv)

We present a novel technique for constructing differentiable order-type operations, including soft ranking, soft top-k selection, and soft permutations. Our approach leverages an efficient closed-form formula for the inverse of the function LapSum, defined as the sum of Laplace distributions. This formulation ensures low computational and memory complexity in selecting the highest activations, enabling losses and gradients to be computed in O(nlogn) time. Through extensive experiments, we demonstrate that our method outperforms state-of-the-art techniques for high-dimensional vectors and large k values. Furthermore, we provide efficient implementations for both CPU and CUDA environments, underscoring the practicality and scalability of our method for large-scale ranking and differentiable ordering problems.