mlsanityData to 14 Oct 2026

Understanding In-Context Learning from Repetitions

arXiv 2310.00297 ↗First posted 30 Sep 2023cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

This paper explores the elusive mechanism underpinning in-context learning in Large Language Models (LLMs). Our work provides a novel perspective by examining in-context learning via the lens of surface repetitions. We quantitatively investigate the role of surface features in text generation, and empirically establish the existence of token co-occurrence reinforcement, a principle that strengthens the relationship between two tokens based on their contextual co-occurrences. By investigating the dual impacts of these features, our research illuminates the internal workings of in-context learning and expounds on the reasons for its failures. This paper provides an essential contribution to the understanding of in-context learning and its potential limitations, providing a fresh perspective on this exciting capability.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 30 Sep 2023.

Builds on

38 of 47 references.

References · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
26 more
  1. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  3. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  4. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  5. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  6. 2022In-context Learning and Induction Headscited by 964
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  8. 2022Transformers learn in-context by gradient descentcited by 732
  9. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  10. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  11. 2019Text Summarization with Pretrained Encoderscited by 556
  12. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  13. 2023Larger language models do in-context learning differentlycited by 394
  14. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 309
  15. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  16. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  17. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  18. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  19. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  20. 2023A Theory of Emergent In-Context Learning as Implicit Structure Inductioncited by 86
  21. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  22. 2016A Simple, Fast Diverse Decoding Algorithm for Neural Generationcited by 58
  23. 2023How Do In-Context Examples Affect Compositional Generalization?cited by 51
  24. 2023Measuring Inductive Biases of In-Context Learning with Underspecified Demonstrationscited by 49
  25. 2021Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherencecited by 45
  26. 2021Straight to the Gradient: Learning to Use Novel Tokens for Neural Text Generationcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 8; 2026: 7.

29 papers.

Sort:

By area:

  1. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 60
  2. 2023WaveCoder: Widespread And Versatile Enhancement For Code Large Language Models By Instruction Tuningcited by 42
  3. 2024The broader spectrum of in-context learningcited by 35
  4. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  5. 2024Repetition Neurons: How Do Language Models Produce Repetitions?cited by 23
  6. 2025S2-MAD: Breaking the Token Barrier to Enhance Multi-Agent Debate Efficiencycited by 22
  7. 2024Prompt-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compressioncited by 18
  8. 2026FIRE-Bench: Evaluating AI Agents on the Rediscovery of Scientific Insightscited by 16
  9. 2025Unveiling Attractor Cycles in Large Language Models: A Dynamical Systems View of Successive Paraphrasingcited by 15
  10. 2025Enhancing Latent Computation in Transformers with Latent Tokenscited by 14
  11. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
  12. 2024RefuteBench: Evaluating Refuting Instruction-Following for Large Language Modelscited by 11
17 more
  1. 2025Repetitions are not all alike: distinct mechanisms sustain repetition in language modelscited by 10
  2. 2026Counterfactual Trace Auditing of LLM Agent Skillscited by 10
  3. 2024Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learningcited by 9
  4. 2024ELICIT: LLM Augmentation via External In-Context Capabilitycited by 7
  5. 2025Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learningcited by 5
  6. 2024From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and Whencited by 3
  7. 2025Task Schema and Binding: A Double Dissociation Study of In-Context Learningcited by 3
  8. 2024Identifying and Analyzing Performance-Critical Tokens in Large Language Modelscited by 2
  9. 2024From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Setcited by 2
  10. 2025Review of Hallucination Understanding in Large Language and Vision Modelscited by 2
  11. 2024Crafting a Good Prompt or Providing Exemplary Dialogues? A Study of In-Context Learning for Persona-based Dialogue Generationcited by 1
  12. 2026Enhancing Large Language Models for Time-Series Forecasting via Vector-Injected In-Context Learningcited by 1
  13. 2024Composited-Nested-Learning with Data Augmentation for Nested Named Entity Recognitioncited by 0
  14. 2026LiFT: How to Enable In-Context Learning for Longitudinal Modellingcited by 0
  15. 2026Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generationcited by 0
  16. 2026Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspectivecited by 0
  17. 2026Not All Answers Are Contextually Persuadable: Inference Dynamics in Large Language Models under Contextual Influencecited by 0