mlsanityData to 7 Oct 2026

When Parts Are Greater Than Sums: Individual LLM Components Can Outperform Full Models

arXiv 2406.13131 ↗First posted 19 Jun 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper studies in-context learning by decomposing the output of large language models into the individual contributions of attention heads and MLPs (components). We observe curious components: good-performing ones that individually do well on a classification task, even when the model performs poorly; bad-performing ones that do much worse than chance; and label-biased components that always predict the same label. We find that component accuracies are well-correlated across different demonstration sets and perturbations of prompt templates. Based on our findings, we propose component reweighting, which learns to linearly re-scale the component activations from a few labeled examples. Given 24 labeled examples, our method improves by an average of 6.0% accuracy points over 24-shot ICL across 8 tasks on Llama-2-7B. Overall, this paper both enriches our understanding of ICL and provides a practical method for improvement by examining model internals.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 64 of its 80 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

64 of 80 references.

References · 64

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
38 more
  1. 2019Root Mean Square Layer Normalizationcited by 1,425
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  7. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  8. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  13. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  14. 2019Are Sixteen Heads Really Better than One?cited by 803
  15. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  16. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  17. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  18. 2019Designing and Interpreting Probes with Control Taskscited by 608
  19. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  20. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  21. 2022Complexity-Based Prompting for Multi-Step Reasoningcited by 387
  22. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  23. 2021True Few-Shot Learning with Language Modelscited by 343
  24. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  25. 2023In-Context Learning Creates Task Vectorscited by 302
  26. 2019Abductive Commonsense Reasoningcited by 300
  27. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  28. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  29. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  30. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  31. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  32. 2023Localizing Model Behavior with Path Patchingcited by 182
  33. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 163
  34. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  35. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 149
  36. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  37. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 126
  38. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 34
  2. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  3. 2025DETAM: Defending LLMs Against Jailbreak Attacks via Targeted Attention Modificationcited by 10
  4. 2025Diversity of Transformer Layers: One Aspect of Parameter Scaling Lawscited by 2
  5. 2025Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queriescited by 1
  6. 2026Domain-Specific Jargon in Large Language Models: A Comparative Analysis between General-Purpose and Specialist Modelscited by 0