mlsanityData to 14 Oct 2026

Benchmarking General-Purpose In-Context Learning

arXiv 2405.17234 ↗First posted 27 May 2024cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In-context learning (ICL) empowers generative models to address new tasks effectively and efficiently on the fly, without relying on any artificially crafted optimization techniques. In this paper, we study extending ICL to address a broader range of tasks with an extended learning horizon and higher improvement potential, namely General Purpose In-Context Learning (GPICL). To this end, we introduce two lightweight benchmarks specifically crafted to train and evaluate GPICL functionalities. Each benchmark encompasses a vast number of tasks characterized by significant task variance. These tasks are also crafted to promote long-horizon in-context learning through continuous generation and interaction, covering domains such as language modeling, decision-making, and world modeling. The benchmarks necessitate the models to leverage contexts and history interactions to enhance their capabilities, which we believe to be the key characteristics of GPICL. Our experiments indicate that the diversity of training tasks is positively correlated with the ability to generalize with ICL, but inversely correlated with zero-shot capabilities. Additionally, our findings indicate that the scale of parameters alone may not be crucial for ICL or GPICL, suggesting alternative approaches such as increasing the scale of contexts and memory states.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 44 of its 55 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

44 of 55 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  7. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  11. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  12. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
32 more
  1. 2020Efficient Transformers: A Surveycited by 790
  2. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  4. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  5. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  6. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  7. 2016Semantic Scene Completion from a Single Depth Imagecited by 472
  8. 2021MetaICL: Learning to Learn In Contextcited by 454
  9. 2023Larger language models do in-context learning differentlycited by 394
  10. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  11. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  12. 2019Neural Legal Judgment Prediction in Englishcited by 209
  13. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  14. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  15. 2022Model-Based Imitation Learning for Urban Drivingcited by 150
  16. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 139
  17. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  18. 2022General-Purpose In-Context Learning by Meta-Learning Transformerscited by 105
  19. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 88
  20. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  21. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
  22. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  23. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 62
  24. 2023XLand-MiniGrid: Scalable Meta-Reinforcement Learning Environments in JAXcited by 55
  25. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 54
  26. 2020Meta Learning Backpropagation And Improving Itcited by 49
  27. 2020Meta-Learning through Hebbian Plasticity in Random Networkscited by 47
  28. 2023Emergence of In-Context Reinforcement Learning from Noise Distillationcited by 27
  29. 2023Brain-inspired learning in artificial neural networks: a reviewcited by 26
  30. 2023In-Context Reinforcement Learning for Variable Action Spacescited by 24
  31. 2024MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Modelscited by 9
  32. 2021Evolving Decomposed Plasticity Rules for Information-Bottlenecked Meta-Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 2.

6 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2025Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worldscited by 4
  3. 2025Context and Diversity Matter: The Emergence of In-Context Learning in World Modelscited by 3
  4. 2026In-Context Optimization for Retrieval-Augmented Generation: A Gradient-Descent Perspectivecited by 2
  5. 2025Meta-Learning Transformers to Improve In-Context Generalizationcited by 0
  6. 2026StateLinFormer: Stateful Training Enhancing Long-term Memory in Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
  2. 2025Probing In-Context Learning: Impact of Task Complexity and Model Architecture on Generalization and Efficiency
  3. 2025Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning
  4. 2023The Mystery of In-Context Learning: A Comprehensive Survey on Interpretation and Analysis
  5. 2024In-Context Language Learning: Architectures and Algorithms
  6. 2023In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
  7. 2024Enhancing In-context Learning via Linear Probe Calibration
  8. 2024Can Custom Models Learn In-Context? An Exploration of Hybrid Architecture Performance on In-Context Learning Tasks