mlsanityData to 14 Oct 2026

In-Context Learning with Many Demonstration Examples

arXiv 2302.04931 ↗First posted 9 Feb 2023cs.CL cs.AIcited by 39

Not on the map. Why?

Abstract (arXiv)

Large pre-training language models (PLMs) have shown promising in-context learning abilities. However, due to the backbone transformer architecture, existing PLMs are bottlenecked by the memory and computational cost when scaling up to a large context size, leaving instruction tuning and in-context learning of many demonstration examples, as well as long-range language modeling under-explored. In this study, we propose a long-range language model EVALM based on an efficient transformer mechanism. EVALM is trained with 8k tokens per batch line and can test up to 256k-lengthed contexts with extrapolation, 128 times to the limit of existing PLMs (e.g. GPT3). Based on EVALM, we scale up the size of examples efficiently in both instruction tuning and in-context learning to explore the boundary of the benefits from more annotated data. Experimental results on a diverse set of tasks show that EVALM achieves 4.1% higher accuracy on average, and the average length of achieving the best accuracy score over tasks is around 12k. We find that in-context learning can achieve higher performance with more demonstrations under many-shot instruction tuning (8k), and further extending the length of instructions (16k) can further improve the upper bound of scaling in-context learning.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 47 of its 60 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

47 of 60 references.

References · 47

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2020Longformer: The Long-Document Transformercited by 3,413
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
35 more
  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  7. 2020Rethinking Attention with Performerscited by 1,601
  8. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  10. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  12. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  13. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  14. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  15. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  16. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  17. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  18. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  19. 2021MetaICL: Learning to Learn In Contextcited by 454
  20. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
  21. 2019Axial Attention in Multidimensional Transformerscited by 358
  22. 2021Random Feature Attentioncited by 354
  23. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  24. 2018Generating Wikipedia by Summarizing Long Sequencescited by 335
  25. 2020Sparse Sinkhorn Attentioncited by 251
  26. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  27. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  28. 2022Learning by Distilling Contextcited by 138
  29. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 132
  30. 2021SOFT: Softmax-free Transformer with Linear Complexitycited by 128
  31. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 63
  32. 2021ABC: Attention with Bounded-memory Controlcited by 36
  33. 2022Prompt Injection: Parameterization of Fixed Inputscited by 36
  34. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  35. 2022CAB: Comprehensive Attention Benchmarking on Long Sequence Modelingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 17; 2025: 13; 2026: 1.

39 papers.

Sort:

By area:

  1. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  2. 2024Many-Shot In-Context Learningcited by 206
  3. 2023Multilingual Machine Translation with Large Language Models: Empirical Results and Analysiscited by 160
  4. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 111
  5. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  6. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  7. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  8. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 50
  9. 2024ICLR: In-Context Learning of Representationscited by 50
  10. 2023SelfEvolve: A Code Evolution Framework via Large Language Modelscited by 43
  11. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 28
  12. 2024A Controlled Study on Long Context Extension and Generalization in LLMscited by 26
27 more
  1. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 21
  2. 2023Efficient Prompting via Dynamic In-Context Learningcited by 18
  3. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  4. 2024Can Many-Shot In-Context Learning Help LLMs as Evaluators? A Preliminary Empirical Studycited by 15
  5. 2024Vector-ICL: In-context Learning with Continuous Vector Representationscited by 13
  6. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  7. 2025More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectivescited by 12
  8. 2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
  9. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  10. 2024Divide, Reweight, and Conquer: A Logit Arithmetic Approach for In-Context Learningcited by 9
  11. 2025What Really Matters in Many-Shot Attacks? An Empirical Study of Long-Context Vulnerabilities in LLMscited by 8
  12. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?cited by 7
  13. 2025Efficient Multi-modal Long Context Learning for Training-free Adaptationcited by 7
  14. 2024Enhancing In-Context Learning via Implicit Demonstration Augmentationcited by 6
  15. 2024Naive Bayes-based Context Extension for Large Language Modelscited by 5
  16. 2023Linear Attention via Orthogonal Memorycited by 4
  17. 2024Enhancing In-Context Learning Performance with just SVD-Based Weight Pruning: A Theoretical Perspectivecited by 4
  18. 2025Where and What Matters: Sensitivity-Aware Task Vectors for Many-Shot Multimodal In-Context Learningcited by 4
  19. 2024Disentangling Questions from Query Generation for Task-Adaptive Retrievalcited by 3
  20. 2023Enhance Reasoning Ability of Visual-Language Models via Large Language Modelscited by 2
  21. 2024Identifying and Analyzing Performance-Critical Tokens in Large Language Modelscited by 2
  22. 2025MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learningcited by 2
  23. 2024In-Context Learning for Long-Context Sentiment Analysis on Infrastructure Project Opinionscited by 1
  24. 2025Compressing Many-Shots in In-Context Learningcited by 1
  25. 2026AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reusecited by 1
  26. 2023Extending Input Contexts of Language Models through Training on Segmented Sequencescited by 0
  27. 2024How to Understand "Support"? An Implicit-enhanced Causal Inference Approach for Weakly-supervised Phrase Groundingcited by 0