mlsanityData to 14 Oct 2026

Attention Sinks and Outliers in Attention Residuals

arXiv 2605.17887 ↗First posted 18 May 2026cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

We propose OASIS, an outlier- and sink-aware method that stabilizes dual-normalized attention-residual architectures through explicit null routing and token-to-depth null coupling. AttnResidual introduces an additional depth-wise normalization channel that improves inter-layer routing flexibility but can also amplify attention sinks, activation outliers, and low-bit quantization error. OASIS builds on explicit Softmax1-based null routes at both the token and depth levels and uses token-level null evidence to downweight depth branches exhibiting stronger null behavior. Theoretically, we characterize a conditional mechanism for sink-like attention concentration under dual normalization, offering insight into the low-bit sensitivity observed in AttnResidual. Experimentally, we compare OASIS against five baselines on three language-model backbones and multiple language-modeling, reasoning, and long-context benchmarks and observe consistent improvements in both attention sink mitigation and post-quantization performance. Notably, relative to Vanilla AttnResidual, OASIS reduces maximum infinity norm by 81.75% and average kurtosis by 95.90%, lowers W8A8 perplexity by 82.00%, and improves W4A4 GSM8K Pass@1 by 42.11% on average across LLaMA-3.2-1B, Qwen3-0.6B, and Phi-4. Code is available at: https://github.com/robinzixuan/OASIS.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 54 of its 62 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

54 of 62 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  12. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
38 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  2. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  3. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  4. 2018Universal Transformerscited by 647
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  6. 2024SpinQuant: LLM quantization with learned rotationscited by 471
  7. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  8. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  9. 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 414
  10. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  11. 2019Revealing the Dark Secrets of BERTcited by 301
  12. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  13. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  14. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 258
  15. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 226
  16. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  17. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  18. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 181
  19. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 151
  20. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 141
  21. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  22. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 131
  23. 2025Why do LLMs attend to the first token?cited by 127
  24. 2023OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Modelscited by 114
  25. 2026Attention Residualscited by 65
  26. 2020Robust Quantization: One Model to Rule Them Allcited by 63
  27. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  28. 2024Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantizationcited by 33
  29. 2024Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Modelscited by 27
  30. 2025Softpick: No Attention Sink, No Massive Activations with Rectified Softmaxcited by 22
  31. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMscited by 20
  32. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 20
  33. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 11
  34. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  35. 2025Fast and Low-Cost Genomic Foundation Models via Outlier Removalcited by 8
  36. 2024ST-MoE-BERT: A Spatial-Temporal Mixture-of-Experts Framework for Long-Term Cross-City Mobility Predictioncited by 7
  37. 2026GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Samplingcited by 6
  38. 2026FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoningcited by 5

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Low-Rank Attention Residualscited by 1
  2. 2026Abstention and Noise Filtering: Two Missing Primitives of Softmax Attentioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
  2. 2026Attention Sinks in VLMs Are Borrowed, Not Seen
  3. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMs
  4. 2024From Attention to Activation: Unravelling the Enigmas of Large Language Models
  5. 2026When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models
  6. 2026SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoders
  7. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapse
  8. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs