SSA: Improving Performance With a Better Scoring Function
Not on the map. Why?
Abstract (arXiv)
While transformer models exhibit strong in-context learning (ICL) abilities, they often fail to generalize under simple distribution shifts. We analyze these failures and identify Softmax, the scoring function in the attention mechanism, as a contributing factor. We propose Scaled Signed Averaging (SSA), a novel attention scoring function that mitigates these failures. SSA significantly improves performance on our ICL tasks and outperforms transformer models with Softmax on several NLP benchmarks and linguistic probing tasks, in both decoder-only and encoder-only architectures.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 21 of its 35 references that are in the map. First posted on arXiv 20 Aug 2025.
Builds on
21 of 35 references.
References · 21
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
- 2022In-context Learning and Induction Headscited by 964
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
- 2022A Survey on In-context Learningcited by 721
- 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
- 2023Trained Transformers Learn Linear Models In-Contextcited by 337
- 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
- 2019Sparse Sequence-to-Sequence Modelscited by 170
- 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
9 more
- 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
- 2023Scan and Snap: Understanding Training Dynamics and Token Composition in 1-layer Transformercited by 102
- 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 88
- 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
- 2022Analyzing Semantic Faithfulness of Language Models via Input Intervention on Question Answeringcited by 6
- 2025Analyzing limits for in-context learningcited by 6
- 2024Re-examining learning linear functions in contextcited by 5
- 2025Self-Adjust Softmaxcited by 3
- 2025LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Modelcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.