mlsanityData to 14 Oct 2026

Many-Shot In-Context Learning

arXiv 2404.11018 ↗First posted 17 Apr 2024cs.LG cs.AI cs.CLcited by 206

Not on the map. Why?

Cited by 27 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) excel at few-shot in-context learning (ICL) -- learning from a few examples provided in context at inference, without any weight updates. Newly expanded context windows allow us to investigate ICL with hundreds or thousands of examples -- the many-shot regime. Going from few-shot to many-shot, we observe significant performance gains across a wide variety of generative and discriminative tasks. While promising, many-shot ICL can be bottlenecked by the available amount of human-generated examples. To mitigate this limitation, we explore two new settings: Reinforced and Unsupervised ICL. Reinforced ICL uses model-generated chain-of-thought rationales in place of human examples. Unsupervised ICL removes rationales from the prompt altogether, and prompts the model only with domain-specific questions. We find that both Reinforced and Unsupervised ICL can be quite effective in the many-shot regime, particularly on complex reasoning tasks. Finally, we demonstrate that, unlike few-shot learning, many-shot learning is effective at overriding pretraining biases, can learn high-dimensional functions with numerical inputs, and performs comparably to fine-tuning. We also find that inference cost increases linearly in the many-shot regime, and frontier LLMs benefit from many-shot ICL to varying degrees. Our analysis also reveals the limitations of next-token prediction loss as an indicator of downstream ICL performance.

Key facts. Cited by 206 papers in the map (data through 14 Oct 2026). Builds on 56 of its 75 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

56 of 75 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
38 more
  1. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  2. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
  3. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  5. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  6. 2022Transformers learn in-context by gradient descentcited by 732
  7. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  8. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  10. 2022Efficiently Scaling Transformer Inferencecited by 539
  11. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  13. 2023Larger language models do in-context learning differentlycited by 394
  14. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  15. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  16. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  17. 2023On the Planning Abilities of Large Language Models : A Critical Investigationcited by 312
  18. 2023In-Context Learning Creates Task Vectorscited by 312
  19. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  20. 2023Effective Long-Context Scaling of Foundation Modelscited by 303
  21. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 257
  22. 2023LEVER: Learning to Verify Language-to-Code Generation with Executioncited by 239
  23. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  24. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  25. 2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 152
  26. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  27. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
  28. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 136
  29. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  30. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  31. 2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
  32. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  33. 2024From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examplescited by 57
  34. 2023In-Context Learning Learns Label Relationships but Is Not Conventional Learningcited by 51
  35. 2022Transformers generalize differently from information stored in context vs in weightscited by 49
  36. 2024Dual Operating Modes of In-Context Learningcited by 43
  37. 2024An Information-Theoretic Analysis of In-Context Learningcited by 40
  38. 2023In-Context Learning with Many Demonstration Examplescited by 39

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 63; 2025: 96; 2026: 47.

206 papers.

Sort:

By area:

  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  4. 2025Agentic Context Engineering: Evolving Contexts for Self-Improving Language Modelscited by 398
  5. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  6. 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 148
  7. 2024CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQLcited by 138
  8. 2025Memento: Fine-tuning LLM Agents without Fine-tuning LLMscited by 127
  9. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  10. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAGcited by 123
  11. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  12. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
38 more
  1. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  2. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  3. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  4. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  5. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  6. 2025ChunkKV: Semantic-Preserving KV Cache Compression for Efficient Long-Context LLM Inferencecited by 60
  7. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 57
  8. 2024Beyond Examples: High-level Automated Reasoning Paradigm in In-Context Learning via MCTScited by 54
  9. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 47
  10. 2025How Do Large Language Monkeys Get Their Power (Laws)?cited by 43
  11. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  12. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  13. 2024The broader spectrum of in-context learningcited by 35
  14. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
  15. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  16. 2024Teach Better or Show Smarter? On Instructions and Exemplars in Automatic Prompt Optimizationcited by 29
  17. 2024LLMs Are In-Context Bandit Reinforcement Learnerscited by 29
  18. 2025Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steeringcited by 29
  19. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  20. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 24
  21. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  22. 2024Retrieval-Augmented Decision Transformer: External Memory for In-context RLcited by 21
  23. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 21
  24. 2024Estimating the Hallucination Rate of Generative AIcited by 20
  25. 2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 20
  26. 2024Is In-Context Learning Sufficient for Instruction Following in LLMs?cited by 19
  27. 2024IDs for AI Systemscited by 19
  28. 2024Exploring and Benchmarking the Planning Capabilities of Large Language Modelscited by 19
  29. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  30. 2024Bayesian scaling laws for in-context learningcited by 19
  31. 2024Toward Understanding In-context vs. In-weight Learningcited by 19
  32. 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Taskscited by 19
  33. 2025The Elicitation Game: Evaluating Capability Elicitation Techniquescited by 18
  34. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  35. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  36. 2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 17
  37. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  38. 2025InfiniteICL: Breaking the Limit of Context Window Size via Long Short-term Memory Transformationcited by 17

50 most cited of 206 All 206 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives
  2. 2025Compressing Many-Shots in In-Context Learning
  3. 2023Self-ICL: Zero-Shot In-Context Learning with Self-Generated Demonstrations
  4. 2026FewMMBench: A Benchmark for Multimodal Few-Shot Learning
  5. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  6. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?
  7. 2026Many-Shot CoT-ICL: Making In-Context Learning Truly Learn
  8. 2024Focused Large Language Models are Stable Many-Shot Learners