Data to 7 Oct 2026

In-Context Learning with Long-Context Models: An In-Depth Exploration

arXiv 2405.00200 ↗First posted 30 Apr 2024cs.CLcited by 126

Not on the map. Why?

Abstract (arXiv)

As model context lengths continue to increase, the number of demonstrations that can be provided in-context approaches the size of entire training datasets. We study the behavior of in-context learning (ICL) at this extreme scale on multiple datasets and models. We show that, for many datasets with large label spaces, performance continues to increase with thousands of demonstrations. We contrast this with example retrieval and finetuning: example retrieval shows excellent performance at low context lengths but has diminished gains with more demonstrations; finetuning is more data hungry than ICL but can exceed long-context ICL performance with additional data. We use the ICL setting to study several properties of both in-context learning and long-context models. We show that long-context ICL is less sensitive to random input shuffling than short-context ICL, that grouping of same-label examples negatively impacts performance, and that the performance boosts do not arise from cumulative gain from encoding many examples together. We conclude that long-context ICL can be an effective tool, and may not require long-context for encoding the demonstration set at all.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 43 of its 69 references that are in the map. First posted on arXiv 30 Apr 2024.

Builds on

43 of 69 references.

References · 43

2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
2022Transformers learn in-context by gradient descentcited by 714
2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
2021MetaICL: Learning to Learn In Contextcited by 450

31 more

2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
2021True Few-Shot Learning with Language Modelscited by 343
2023In-Context Learning Creates Task Vectorscited by 302
2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 231
2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
2023What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learningcited by 149
2023Emergent and Predictable Memorization in Large Language Modelscited by 147
2022Diverse Demonstrations Improve In-context Compositional Generalizationcited by 136
2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
2023LM-Infinite: Zero-Shot Extreme Length Generalization for Large Language Modelscited by 120
2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
2022Parallel Context Windows for Large Language Modelscited by 81
2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
2023BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfercited by 68
2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
2023Coverage-based Example Selection for In-Context Learningcited by 58
2023Towards Understanding In-Context Learning with Contrastive Demonstrations and Saliency Mapscited by 35
2023In-Context Learning for Text Classification with Many Labelscited by 35
2019Mining Discourse Markers for Unsupervised Sentence Representation Learningcited by 34
2024Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMscited by 30
2022Prompt-Augmented Linear Probing: Scaling beyond the Limit of Few-shot In-Context Learnerscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 39; 2025: 60; 2026: 27.

126 papers.

Sort:

By area:

2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMscited by 142
2024How to Train Long-Context Language Models (Effectively)cited by 141
2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 64
2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 39
2024The broader spectrum of in-context learningcited by 33
2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 33
2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28

38 more

2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27
2024A Controlled Study on Long Context Extension and Generalization in LLMscited by 26
2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 26
2024Private prediction for large-scale synthetic text generationcited by 24
2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 20
2024Is In-Context Learning Sufficient for Instruction Following in LLMs?cited by 19
2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
2024AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrationscited by 19
2025NExtLong: Toward Effective Long-Context Training without Long Documentscited by 19
2024FlashMask: Efficient and Rich Mask Extension of FlashAttentioncited by 18
2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Taskscited by 17
2024Revisiting In-context Learning Inference Circuit in Large Language Modelscited by 16
2024Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorizationcited by 15
2024Can we teach language models to gloss endangered languages?cited by 14
2025The Geometry of Prompting: Unveiling Distinct Mechanisms of Task Adaptation in Language Modelscited by 14
2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
2024A Fused Large Language Model for Predicting Startup Successcited by 13
2024Vector-ICL: In-context Learning with Continuous Vector Representationscited by 13
2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
2025More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectivescited by 12
2025Problem-Solving Logic Guided Curriculum In-Context Learning for LLMs Complex Reasoningcited by 12
2025Efficient Many-Shot In-Context Learning with Dynamic Block-Sparse Attentioncited by 12
2025A Survey on Transformer Context Extension: Approaches and Evaluationcited by 11
2024On Many-Shot In-Context Learning for Long-Context Evaluationcited by 10
2025A Survey of Scaling in Large Language Model Reasoningcited by 10
2025Focus Directions Make Your Language Models Pay More Attention to Relevant Contextscited by 9
2024Focused Large Language Models are Stable Many-Shot Learnerscited by 8
2024Differential Transformercited by 8
2024A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classificationcited by 8
2025Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimationcited by 8
2025Prompt-MII: Meta-Learning Instruction Induction for LLMscited by 8
2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 8
2024ELICIT: LLM Augmentation via External In-Context Capabilitycited by 7
2024FM2DS: Few-Shot Multimodal Multihop Data Synthesis with Knowledge Distillation for Question Answeringcited by 7
2024Revisiting In-Context Learning with Long Context Language Modelscited by 7
2025MIR-Bench: Can Your LLM Recognize Complicated Patterns via Many-Shot In-Context Reasoning?cited by 7

50 most cited of 126 All 126 (JSON)