mlsanityData to 14 Oct 2026

Softpick: No Attention Sink, No Massive Activations with Rectified Softmax

arXiv 2504.20966 ↗First posted 29 Apr 2025cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our experiments with 340M and 1.8B parameter models demonstrate that softpick achieves 0% sink rate consistently. The softpick transformers produce hidden states with significantly lower kurtosis and creates sparse attention maps. Quantized models using softpick outperform softmax on standard benchmarks, with a particularly pronounced advantage at lower bit precisions. Our analysis and discussion shows how softpick has the potential to open new possibilities for quantization, low-precision training, sparsity optimization, pruning, and interpretability. Our code: https://github.com/zaydzuhri/softpick-attention

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

35 of 59 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2016Pointer Sentinel Mixture Modelscited by 3,096
  8. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  9. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  11. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  12. 2020GLU Variants Improve Transformercited by 1,804
23 more
  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  2. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  3. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  4. 2020Quantifying Attention Flow in Transformerscited by 775
  5. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  6. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 234
  7. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  8. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  9. 2025Why do LLMs attend to the first token?cited by 127
  10. 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
  11. 2023FP8-LM: Training FP8 Large Language Modelscited by 77
  12. 2024Scaling FP8 training to trillion-token LLMscited by 65
  13. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 56
  14. 2024Training-Free Activation Sparsity in Large Language Modelscited by 55
  15. 2025Scalable-Softmax Is Superior for Attentioncited by 45
  16. 2023Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edgecited by 33
  17. 2025BitNet v2: Native 4-bit Activations with Hadamard Transformation for 1-bit LLMscited by 18
  18. 2024LongHeads: Multi-Head Attention is Secretly a Long Context Processorcited by 16
  19. 2025Analysis of Attention in Video Diffusion Transformerscited by 15
  20. 2021Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modelingcited by 14
  21. 2024From Attention to Activation: Unravelling the Enigmas of Large Language Modelscited by 13
  22. 2025A Refined Analysis of Massive Activations in LLMscited by 12
  23. 2024XLSR-Transducer: Streaming ASR for Self-Supervised Pretrained Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 17.

22 papers.

Sort:

By area:

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  2. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  3. 2025Unveiling Super Experts in Mixture-of-Experts Large Language Modelscited by 17
  4. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 11
  5. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  6. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11
  7. 2025Hidden Dynamics of Massive Activations in Transformer Trainingcited by 10
  8. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 6
  9. 2026Threshold Differential Attention for Sink-Free, Ultra-Sparse, and Non-Dispersive Language Modelingcited by 5
  10. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapsecited by 5
  11. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  12. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 4
10 more
  1. 2026EarlyTom: Early Token Compression Completes Fast Video Understandingcited by 4
  2. 2026Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformerscited by 3
  3. 2026A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Modelscited by 2
  4. 2026Attention Sinks and Outliers in Attention Residualscited by 2
  5. 2026FLUID: Continuous-Time Hyperconnected Sparse Transformer for Sink-Free Learningcited by 1
  6. 2026Disentangling Semantic Attention from Structural Bias in the Attention Manifoldcited by 0
  7. 2026From Attention Masks to Inert Zero-Vector Tokens: OAttention and O-Closure for Token Dynamicscited by 0
  8. 2026Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?cited by 0
  9. 2026Abstention and Noise Filtering: Two Missing Primitives of Softmax Attentioncited by 0
  10. 2026Underscoring the Problem: Why Softpick Fails at Initializationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Pretraining Transformers with Quantized Softmax in Attention
  2. 2026EFQ-Softmax: Exp-Free Quantization for Softmax
  3. 2026From Attention Sensitivity to Layer Role: Revisiting Mixed-Precision Quantization of Transformers
  4. 2025Exploiting Information Redundancy in Attention Maps for Extreme Quantization of Vision Transformers
  5. 2026BinaryAttention: One-Bit QK-Attention for Vision and Diffusion Transformers
  6. 2024Exploring the Frontiers of Softmax: Provable Optimization, Applications in Diffusion Model, and Beyond
  7. 2026Linearizing Vision Transformer with Test-Time Training
  8. 2026BAPS: A Fine-Grained Low-Precision Scheme for Softmax in Attention via Block-Aware Precision reScaling