mlsanityData to 14 Oct 2026

Multi-Layer Attention is the Amplifier of Demonstration Effectiveness

arXiv 2508.00385 ↗First posted 1 Aug 2025cs.CL cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Numerous studies have investigated the underlying mechanisms of in-context learning (ICL) effectiveness to inspire the design of related methods. However, existing work predominantly assumes the effectiveness of the demonstrations provided within ICL, while many research indicates that not all demonstrations are effective, failing to yielding any performance improvement during ICL. Therefore, in this paper, we investigate the reasons behind demonstration ineffectiveness. Our analysis is based on gradient flow and linear self-attention models. By setting the gradient flow to zero, we deduce that a demonstration becomes ineffective if its information has either been learned by the model or is irrelevant to the user query. Furthermore, we demonstrate that in multi-layer models, the disparity in effectiveness among demonstrations is amplified with layer increasing, causing the model to focus more on effective ones. Considering that current demonstration selection methods primarily focus on the relevance to the user query while overlooking the information that the model has already assimilated, we propose a novel method called GradS, which leverages gradient flow for demonstration selection. We use the magnitude of the gradient flow of the demonstration with respect to a given user query as the criterion, thereby ensuring the effectiveness of the chosen ones. We validate our derivation and GradS on four prominent LLMs across five mainstream datasets. The experimental results confirm that the disparity in effectiveness among demonstrations is magnified as the model layer increases, substantiating our derivations. Moreover, GradS achieves a relative improvement of 6.8% on average over the strongest baselines, demonstrating its effectiveness.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 52 of its 64 references that are in the map. First posted on arXiv 1 Aug 2025.

Builds on

52 of 64 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  10. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  11. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  12. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
38 more
  1. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  2. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  3. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
  4. 2023Unified Demonstration Retriever for In-Context Learningcited by 142
  5. 2023The Learnability of In-Context Learningcited by 127
  6. 2022Complementary Explanations for Effective In-Context Learningcited by 71
  7. 2019Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answeringcited by 59
  8. 2024Revisiting Demonstration Selection Strategies in In-Context Learningcited by 58
  9. 2024Competition Dynamics Shape Algorithmic Phases of In-Context Learningcited by 54
  10. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  11. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 49
  12. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 46
  13. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  14. 2024Dual Operating Modes of In-Context Learningcited by 43
  15. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrationscited by 42
  16. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  17. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  18. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysiscited by 34
  19. 2025Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thoughtcited by 32
  20. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  21. 2024Linear Transformers are Versatile In-Context Learnerscited by 28
  22. 2024Differential learning kinetics govern the transition from memorization to generalization during in-context learningcited by 21
  23. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  24. 2025Strategy Coopetition Explains the Emergence and Transience of In-Context Learningcited by 17
  25. 2025When can in-context learning generalize out of task distribution?cited by 16
  26. 2024In-Context Learning State Vector with Inner and Momentum Optimizationcited by 15
  27. 2024How Transformers Utilize Multi-Head Attention in In-Context Learning? A Case Study on Sparse Linear Regressioncited by 14
  28. 2023In-Context Learning Dynamics with Random Binary Sequencescited by 13
  29. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  30. 2024Demonstration Augmentation for Zero-shot In-context Learningcited by 11
  31. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learningcited by 8
  32. 2024Demonstration Selection for In-Context Learning via Reinforcement Learningcited by 8
  33. 2024ELICIT: LLM Augmentation via External In-Context Capabilitycited by 7
  34. 2025In-context denoising with one-layer transformers: connections between attention and associative memory retrievalcited by 7
  35. 2024Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Modelscited by 4
  36. 2025Understanding the Generalization of In-Context Learning in Transformers: An Empirical Studycited by 4
  37. 2022Maximum Batch Frobenius Norm for Multi-Domain Text Classificationcited by 3
  38. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025TASP: Topology-aware Sequence Parallelismcited by 0