mlsanityData to 14 Oct 2026

Sparse Sequence-to-Sequence Models

arXiv 1905.05702 ↗First posted 14 May 2019cs.CL cs.LGcited by 170

Not on the map. Why?

Abstract (arXiv)

Sequence-to-sequence models are a powerful workhorse of NLP. Most variants employ a softmax transformation in both their attention mechanism and output layer, leading to dense alignments and strictly positive output probabilities. This density is wasteful, making models less interpretable and assigning probability mass to many implausible outputs. In this paper, we propose sparse sequence-to-sequence models, rooted in a new family of α-entmax transformations, which includes softmax and sparsemax as particular cases, and is sparse for any α > 1. We provide fast algorithms to evaluate these transformations and their gradients, which scale well for large vocabulary sizes. Our models are able to produce sparse alignments and to assign nonzero probability to a short list of plausible outputs, sometimes rendering beam search exact. Experiments on morphological inflection and machine translation reveal consistent gains over dense models.

Key facts. Cited by 170 papers in the map (data through 14 Oct 2026). First posted on arXiv 14 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 28; 2022: 20; 2023: 16; 2024: 25; 2025: 28; 2026: 53.

170 papers.

Sort:

By area:

  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  2. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263
  3. 2021Word Alignment by Fine-tuning Embeddings on Parallel Corporacited by 186
  4. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  5. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  6. 2021NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learningcited by 88
  7. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  8. 2021Language Model Evaluation Beyond Perplexitycited by 78
  9. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  10. 2023Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformerscited by 67
  11. 2021Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translationcited by 65
  12. 2022NFormer: Robust Person Re-identification with Neighbor Transformercited by 57
38 more
  1. 2023A Study on ReLU and Softmax in Transformercited by 53
  2. 2021A Survey on Green Deep Learningcited by 52
  3. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 52
  4. 2021DANets: Deep Abstract Networks for Tabular Data Classification and Regressioncited by 50
  5. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 46
  6. 2021Boost then Convolve: Gradient Boosting Meets Graph Neural Networkscited by 44
  7. 2023On Sparse Modern Hopfield Modelcited by 42
  8. 2021Sparse Attention with Linear Unitscited by 41
  9. 2023STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Predictioncited by 35
  10. 2022SIMPLE: A Gradient Estimator for k-Subset Samplingcited by 34
  11. 2022A Measure-Theoretic Characterization of Tight Language Modelscited by 33
  12. 2021Is Sparse Attention more Interpretable?cited by 31
  13. 2021ARM-Net: Adaptive Relation Modeling Network for Structured Datacited by 24
  14. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  15. 2024BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Modelcited by 21
  16. 2025Long-Context Generalization with Sparse Attentioncited by 21
  17. 2022Learning to Scaffold: Optimizing Model Explanations for Teachingcited by 20
  18. 2023From Latent Graph to Latent Topology Inference: Differentiable Cell Complex Modulecited by 19
  19. 2023Compact Neural Graphics Primitives with Learned Hash Probingcited by 19
  20. 2024Towards Robust Knowledge Tracing Models via k-Sparse Attentioncited by 19
  21. 2024Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codescited by 19
  22. 2023On the Efficacy of Sampling Adapterscited by 17
  23. 2022Data-Efficient and Interpretable Tabular Anomaly Detectioncited by 15
  24. 2021Sparse Continuous Distributions and Fenchel-Young Lossescited by 14
  25. 2021Predicting Attention Sparsity in Transformerscited by 13
  26. 2024Sparse and Structured Hopfield Networkscited by 13
  27. 2022Knowledge Infused Decodingcited by 12
  28. 2025AdaSplash: Adaptive Sparse Flash Attentioncited by 12
  29. 2021Smoothing and Shrinking the Sparse Seq2Seq Search Spacecited by 11
  30. 2021Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modelingcited by 11
  31. 2023GradTree: Learning Axis-Aligned Decision Trees with Gradient Descentcited by 11
  32. 2023GRANDE: Gradient-Based Decision Tree Ensembles for Tabular Datacited by 11
  33. 2023Joint Sensing, Communication, and AI: A Trifecta for Resilient THz User Experiencescited by 10
  34. 2025Loss Functions and Operators Generated by f-Divergencescited by 10
  35. 2022GANDALF: Gated Adaptive Network for Deep Automated Learning of Featurescited by 9
  36. 2024Language Generation with Strictly Proper Scoring Rulescited by 8
  37. 2024MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shiftscited by 8
  38. 2025Scaled-Dot-Product Attention as One-Sided Entropic Optimal Transportcited by 8

50 most cited of 170 All 170 (JSON)