mlsanityData to 14 Oct 2026

On Many-Shot In-Context Learning for Long-Context Evaluation

arXiv 2411.07130 ↗First posted 11 Nov 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

Many-shot in-context learning (ICL) has emerged as a unique setup to both utilize and test the ability of large language models to handle long context. This paper delves into long-context language model (LCLM) evaluation through many-shot ICL. We first ask: what types of ICL tasks benefit from additional demonstrations, and how effective are they in evaluating LCLMs? We find that classification and summarization tasks show performance improvements with additional demonstrations, while translation and reasoning tasks do not exhibit clear trends. Next, we investigate the extent to which different tasks necessitate retrieval versus global context understanding. We develop metrics to categorize ICL tasks into two groups: (i) similar-sample learning (SSL): tasks where retrieval of the most similar examples is sufficient for good performance, and (ii) all-sample learning (ASL): tasks that necessitate a deeper comprehension of all examples in the prompt. Lastly, we introduce a new many-shot ICL benchmark, MANYICLBENCH, to characterize model's ability on both fronts and benchmark 12 LCLMs using MANYICLBENCH. We find that while state-of-the-art models demonstrate good performance up to 64k tokens in SSL tasks, many models experience significant performance drops at only 16k tokens in ASL tasks.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 34 of its 45 references that are in the map. First posted on arXiv 11 Nov 2024.

Builds on

34 of 45 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
22 more
  1. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  3. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  4. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  6. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  7. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  8. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  9. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  10. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  11. 2023Adapting Language Models to Compress Contextscited by 356
  12. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  13. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 313
  14. 2024Long-context LLMs Struggle with Long In-context Learningcited by 286
  15. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  16. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  17. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  18. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  19. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  20. 2020Dialogue-Based Relation Extractioncited by 68
  21. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 50
  22. 2024Dual Operating Modes of In-Context Learningcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  2. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  3. 2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?cited by 7
  4. 2025MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretrainingcited by 3
  5. 2026SynAE: A Framework for Measuring the Quality of Synthetic Data for Tool-Calling Agent Evaluationscited by 2
  6. 2026MARCH: Scaling Recurrent Memory with Content-Routed State Anchorscited by 2
  7. 2026AdapShot: Adaptive Many-Shot In-Context Learning with Semantic-Aware KV Cache Reusecited by 1
  8. 2026CALVO: Improve Serving Efficiency for LLM Inferences with Intense Network Demandscited by 0
  9. 2026Context Memorization for Efficient Long Context Generationcited by 0
  10. 2026Unifying Conformal Language Tasks with In-Context Ensemblescited by 0