mlsanityData to 7 Oct 2026

Sparse Sinkhorn Attention

arXiv 2002.11296 ↗First posted 26 Feb 2020cs.LG cs.CLcited by 250

Not on the map. Why?

Abstract (arXiv)

We propose Sparse Sinkhorn Attention, a new efficient and sparse method for learning to attend. Our method is based on differentiable sorting of internal representations. Concretely, we introduce a meta sorting network that learns to generate latent permutations over sequences. Given sorted sequences, we are then able to compute quasi-global attention with only local windows, improving the memory efficiency of the attention module. To this end, we propose new algorithmic innovations such as Causal Sinkhorn Balancing and SortCut, a dynamic sequence truncation method for tailoring Sinkhorn Attention for encoding and/or decoding purposes. Via extensive experiments on algorithmic seq2seq sorting, language modeling, pixel-wise image generation, document classification and natural language inference, we demonstrate that our memory efficient Sinkhorn Attention method is competitive with vanilla attention and consistently outperforms recently proposed efficient Transformer models such as Sparse Transformers.

Key facts. Cited by 250 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 51; 2022: 49; 2023: 49; 2024: 41; 2025: 37; 2026: 23.

250 papers.

Sort:

By area:

  1. 2022FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecastingcited by 1,200
  2. 2021Transformers in Vision: A Surveycited by 1,009
  3. 2021Multiscale Vision Transformerscited by 896
  4. 2021CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windowscited by 526
  5. 2021A Survey of Transformerscited by 399
  6. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  7. 2021Random Feature Attentioncited by 350
  8. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  9. 2021Efficient Attentions for Long Document Summarizationcited by 342
  10. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  11. 2021Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformerscited by 237
  12. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
38 more
  1. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  2. 2022cosFormer: Rethinking Softmax in Attentioncited by 212
  3. 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
  4. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  6. 2023Efficient Large Language Models: A Surveycited by 183
  7. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  8. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  9. 2021Learned Token Pruning for Transformerscited by 138
  10. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  11. 2022Liquid Structural State-Space Modelscited by 134
  12. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  13. 2022Vision-Centric BEV Perception: A Surveycited by 126
  14. 2023SparQ Attention: Bandwidth-Efficient LLM Inferencecited by 119
  15. 2023Retrieval meets Long Context Large Language Modelscited by 114
  16. 2022Learning Vector-Quantized Item Representation for Transferable Sequential Recommenderscited by 112
  17. 2021An Attention Free Transformercited by 111
  18. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  19. 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
  20. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systemscited by 101
  21. 2022The Devil in Linear Transformercited by 98
  22. 2021Luna: Linear Unified Nested Attentioncited by 96
  23. 2021Finetuning Pretrained Transformers into RNNscited by 94
  24. 2021Learning to Match Features with Seeded Graph Matching Networkcited by 92
  25. 2021Sparse is Enough in Scaling Transformerscited by 86
  26. 2023Diffusion Models Without Attentioncited by 75
  27. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  28. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  29. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  30. 2022Flowformer: Linearizing Transformers with Conservation Flowscited by 69
  31. 2021Token Pooling in Vision Transformerscited by 68
  32. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64
  33. 2021Summ^N: A Multi-Stage Summarization Framework for Long Input Dialogues and Documentscited by 64
  34. 2022Single Stage Virtual Try-on via Deformable Attention Flowscited by 64
  35. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  36. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  37. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  38. 2021Memory-efficient Transformers via Top-k Attentioncited by 59

50 most cited of 250 All 250 (JSON)