mlsanityData to 7 Oct 2026

From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

arXiv 1602.02068 ↗First posted 5 Feb 2016cs.CL cs.LG stat.MLcited by 337

Not on the map. Why?

Abstract (arXiv)

We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. After deriving its properties, we show how its Jacobian can be efficiently computed, enabling its use in a network trained with backpropagation. Then, we propose a new smooth and convex loss function which is the sparsemax analogue of the logistic loss. We reveal an unexpected connection between this new loss and the Huber classification loss. We obtain promising empirical results in multi-label classification problems and in attention-based neural networks for natural language inference. For the latter, we achieve a similar performance as the traditional softmax, but with a selective, more compact, attention focus.

Key facts. Cited by 337 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Feb 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 56; 2022: 54; 2023: 46; 2024: 52; 2025: 57; 2026: 71.

337 papers.

Sort:

By area:

  1. 2021Deep Neural Networks and Tabular Data: A Surveycited by 489
  2. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  3. 2021Trustworthy AI: From Principles to Practicescited by 206
  4. 2022CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Taskcited by 205
  5. 2021Efficient and Modular Implicit Differentiationcited by 199
  6. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  7. 2023Decision-Focused Learning: Foundations, State of the Art, Benchmark and Future Opportunitiescited by 129
  8. 2022Truncation Sampling as Language Model Desmoothingcited by 121
  9. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 114
  10. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  11. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  12. 2023Conditional Adapters: Parameter-efficient Transfer Learning with Fast Inferencecited by 69
38 more
  1. 2021Iterative Methods for Private Synthetic Data: Unifying Framework and New Methodscited by 58
  2. 2025Projecting Assumptions: The Duality Between Sparse Autoencoders and Concept Geometrycited by 58
  3. 2023A Study on ReLU and Softmax in Transformercited by 53
  4. 2024MADTP: Multimodal Alignment-Guided Dynamic Token Pruning for Accelerating Vision-Language Transformercited by 53
  5. 2021A Survey on Green Deep Learningcited by 52
  6. 2024Sparser is Faster and Less is More: Efficient Sparse Attention for Long-Range Transformerscited by 51
  7. 2021Differentially Private Query Release Through Adaptive Projectioncited by 50
  8. 2024Softmax is not Enough (for Sharp Size Generalisation)cited by 42
  9. 2021Sparse Attention with Linear Unitscited by 41
  10. 2022T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interactioncited by 41
  11. 2023On Sparse Modern Hopfield Modelcited by 41
  12. 2024Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Modelscited by 38
  13. 2021Understanding Interlocking Dynamics of Cooperative Rationalizationcited by 36
  14. 2023SiDA-MoE: Sparsity-Inspired Data-Aware Serving for Efficient and Scalable Large Mixture-of-Experts Modelscited by 36
  15. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 36
  16. 2023Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokenscited by 35
  17. 2023STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Predictioncited by 35
  18. 2022SIMPLE: A Gradient Estimator for k-Subset Samplingcited by 34
  19. 2022A Measure-Theoretic Characterization of Tight Language Modelscited by 32
  20. 2021Is Sparse Attention more Interpretable?cited by 31
  21. 2024Transformers Can Represent n-gram Language Modelscited by 31
  22. 2021Explain and Predict, and then Predict Againcited by 29
  23. 2022Learning with Combinatorial Optimization Layers: a Probabilistic Approachcited by 28
  24. 2022SEAT: Stable and Explainable Attentioncited by 25
  25. 2021ARM-Net: Adaptive Relation Modeling Network for Structured Datacited by 24
  26. 2021Sampling Network Guided Cross-Entropy Method for Unsupervised Point Cloud Registrationcited by 24
  27. 2022Alternating Differentiation for Optimization Layerscited by 22
  28. 2022DABERT: Dual Attention Enhanced BERT for Semantic Matchingcited by 21
  29. 2023Dual Path Modeling for Semantic Matching by Perceiving Subtle Conflictscited by 21
  30. 2021(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performancecited by 20
  31. 2022Learning to Scaffold: Optimizing Model Explanations for Teachingcited by 20
  32. 2021SPECTRA: Sparse Structured Text Rationalizationcited by 19
  33. 2022MPCViT: Searching for Accurate and Efficient MPC-Friendly Vision Transformer with Heterogeneous Attentioncited by 19
  34. 2023Compact Neural Graphics Primitives with Learned Hash Probingcited by 19
  35. 2024Self-attention Networks Localize When QK-eigenspectrum Concentratescited by 19
  36. 2024Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Actioncited by 19
  37. 2021BERTTune: Fine-Tuning Neural Machine Translation with BERTScorecited by 18
  38. 2024Towards Robust Knowledge Tracing Models via k-Sparse Attentioncited by 18

50 most cited of 337 All 337 (JSON)