mlsanityData to 7 Oct 2026

MultiMax: Sparse and Multi-Modal Attention Learning

arXiv 2406.01189 ↗First posted 3 Jun 2024cs.LG cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

SoftMax is a ubiquitous ingredient of modern machine learning algorithms. It maps an input vector onto a probability simplex and reweights the input by concentrating the probability mass at large entries. Yet, as a smooth approximation to the Argmax function, a significant amount of probability mass is distributed to other, residual entries, leading to poor interpretability and noise. Although sparsity can be achieved by a family of SoftMax variants, they often require an alternative loss function and do not preserve multi-modality. We show that this trade-off between multi-modality and sparsity limits the expressivity of SoftMax as well as its variants. We provide a solution to this tension between objectives by proposing a piece-wise differentiable function, termed MultiMax, which adaptively modulates the output distribution according to input entry range. Through comprehensive analysis and evaluation, we show that MultiMax successfully produces a distribution that supresses irrelevant entries while preserving multimodality, with benefits in image classification, language modeling and machine translation. The code is available at https://github.com/ZhouYuxuanYX/MultiMax.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 45 of its 63 references that are in the map. First posted on arXiv 3 Jun 2024.

Builds on

45 of 63 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  3. 2017Graph Attention Networkscited by 8,854
  4. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  11. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  12. 2016Deep Networks with Stochastic Depthcited by 973
33 more
  1. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  2. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  3. 2020Quantifying Attention Flow in Transformerscited by 759
  4. 2021Going deeper with Image Transformerscited by 759
  5. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  6. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  7. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 544
  8. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  9. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  10. 2019Sparse Sequence-to-Sequence Modelscited by 167
  11. 2017On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learningcited by 136
  12. 2022Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practicecited by 136
  13. 2017Structured Attention Networkscited by 95
  14. 2023System 2 Attention (is something you might need too)cited by 85
  15. 2020Bayesian Graph Neural Networks with Adaptive Connection Samplingcited by 81
  16. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 71
  17. 2020Understanding Anomaly Detection with Deep Invertible Networks through Hierarchies of Distributions and Featurescited by 64
  18. 2021KVT: k-NN Attention for Boosting Vision Transformerscited by 64
  19. 2020RealFormer: Transformer Likes Residual Attentioncited by 62
  20. 2019Explicit Sparse Transformer: Concentrated Attention Through Explicit Selectioncited by 61
  21. 2019Selective Attention for Context-aware Neural Machine Translationcited by 60
  22. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  23. 2020Talking-Heads Attentioncited by 57
  24. 2016A Convolutional Encoder Model for Neural Machine Translationcited by 54
  25. 2021Vision Transformers with Patch Diversificationcited by 45
  26. 2017A Regularized Framework for Sparse and Structured Neural Attentioncited by 38
  27. 2018Latent Alignment and Variational Attentioncited by 36
  28. 2022Hypergraph Transformer for Skeleton-based Action Recognitioncited by 36
  29. 2018On Controllable Sparse Alternatives to Softmaxcited by 26
  30. 2020Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoderscited by 15
  31. 2022SP-ViT: Learning 2D Spatial Priors for Vision Transformerscited by 12
  32. 2020On Sparsifying Encoder Outputs in Sequence-to-Sequence Modelscited by 8
  33. 2021Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  2. 2025MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scorescited by 4
  3. 2026SeedPolicy: Horizon Scaling via Self-Evolving Diffusion Policy for Robot Manipulationcited by 4
  4. 2026Accurate and Efficient Multi-Channel Time Series Forecasting via Sparse Attention Mechanismcited by 0
  5. 2026Sparse Attention for Dense Open-Vocabulary Prediction in CLIPcited by 0