mlsanityData to 7 Oct 2026

Long-context LLMs Struggle with Long In-context Learning

arXiv 2404.02060 ↗First posted 2 Apr 2024cs.CL cs.AIcited by 285

Not on the map. Why?

Cited by 42 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) have made significant strides in handling long sequences. Some models like Gemini could even to be capable of dealing with millions of tokens. However, their performance evaluation has largely been confined to metrics like perplexity and synthetic tasks, which may not fully capture their true abilities in more challenging, real-world scenarios. We introduce a benchmark (LongICLBench) for long in-context learning in extreme-label classification using six datasets with 28 to 174 classes and input lengths from 2K to 50K tokens. Our benchmark requires LLMs to comprehend the entire input to recognize the massive label spaces to make correct predictions. We evaluate on 15 long-context LLMs and find that they perform well on less challenging classification tasks with smaller label space and shorter demonstrations. However, they struggle with more challenging task like Discovery with 174 labels, suggesting a gap in their ability to process long, context-rich sequences. Further analysis reveals a bias towards labels presented later in the sequence and a need for improved reasoning over multiple pieces of information. Our study reveals that long context understanding and reasoning is still a challenging task for the existing LLMs. We believe LongICLBench could serve as a more realistic evaluation for the future long-context LLMs.

Key facts. Cited by 285 papers in the map (data through 7 Oct 2026). Builds on 41 of its 50 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

41 of 50 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
29 more
  1. 2024Yi: Open Foundation Models by 01.AIcited by 706
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  4. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
  5. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  6. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  7. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  8. 2024InternLM2 Technical Reportcited by 391
  9. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  10. 2024LongRoPE: Extending LLM Context Window Beyond 2 Million Tokenscited by 302
  11. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  12. 2016SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documentscited by 245
  13. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  14. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 206
  15. 2022Exploring Length Generalization in Large Language Modelscited by 204
  16. 2023Augmenting Language Models with Long-Term Memorycited by 186
  17. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  18. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  19. 2021Few-NERD: A Few-Shot Named Entity Recognition Datasetcited by 150
  20. 2024Same Task, More Tokens: the Impact of Input Length on the Reasoning Performance of Large Language Modelscited by 150
  21. 2023Focused Transformer: Contrastive Training for Context Scalingcited by 133
  22. 2023PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Trainingcited by 109
  23. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  24. 2022Parallel Context Windows for Large Language Modelscited by 81
  25. 2020Dialogue-Based Relation Extractioncited by 68
  26. 2023When does In-context Learning Fall Short and Why? A Study on Specification-Heavy Taskscited by 37
  27. 2019Mining Discourse Markers for Unsupervised Sentence Representation Learningcited by 34
  28. 2021ConvFiT: Conversational Fine-Tuning of Pretrained Language Modelscited by 25
  29. 2024E^2-LLM: Efficient and Extreme Length Extension of Large Language Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 86; 2025: 134; 2026: 65.

285 papers.

Sort:

By area:

  1. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  2. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  3. 2024Chain of Agents: Large Language Models Collaborating on Long-Context Taskscited by 201
  4. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)cited by 170
  5. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  6. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  7. 2024AutoSurvey: Large Language Models Can Automatically Write Surveyscited by 116
  8. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 96
  9. 2024From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Futurecited by 85
  10. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  11. 2024Speculative RAG: Enhancing Retrieval Augmented Generation through Draftingcited by 76
  12. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
38 more
  1. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  2. 2024CompAct: Compressing Retrieved Documents Actively for Question Answeringcited by 64
  3. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  4. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
  5. 2024Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?cited by 53
  6. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  7. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  8. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  9. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 45
  10. 2025DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-trainingcited by 45
  11. 2025WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Researchcited by 44
  12. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 43
  13. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 42
  14. 2024Can LLMs Understand Time Series Anomalies?cited by 41
  15. 2024Don't Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Taskscited by 38
  16. 2026Doc-to-LoRA: Learning to Instantly Internalize Contextscited by 36
  17. 2024Offset Unlearning for Large Language Modelscited by 33
  18. 2025TIMER: Temporal Instruction Modeling and Evaluation for Longitudinal Clinical Recordscited by 33
  19. 2024LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarioscited by 31
  20. 2025TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Reviewcited by 31
  21. 2025Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQLcited by 30
  22. 2025MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughlycited by 29
  23. 2025LongProc: Benchmarking Long-Context Language Models on Long Procedural Generationcited by 28
  24. 2025SurveyX: Academic Survey Automation via Large Language Modelscited by 27
  25. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 26
  26. 2025Agentic AI Security: Threats, Defenses, Evaluation, and Open Challengescited by 26
  27. 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 25
  28. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  29. 2024ADELIE: Aligning Large Language Models on Information Extractioncited by 24
  30. 2024DrHouse: An LLM-empowered Diagnostic Reasoning System through Harnessing Outcomes from Sensor Data and Expert Knowledgecited by 24
  31. 2024Multi-Agent Large Language Models for Conversational Task-Solvingcited by 24
  32. 2024Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlesslycited by 23
  33. 2025CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehensioncited by 23
  34. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  35. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  36. 2024Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learningcited by 21
  37. 2024An LLM Agent for Automatic Geospatial Data Analysiscited by 21
  38. 2024Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projectscited by 21

50 most cited of 285 All 285 (JSON)