mlsanityData to 7 Oct 2026

Unlocking In-Context Learning for Natural Datasets Beyond Language Modelling

arXiv 2501.06256 ↗First posted 9 Jan 2025cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) exhibit In-Context Learning (ICL), which enables the model to perform new tasks conditioning only on the examples provided in the context without updating the model's weights. While ICL offers fast adaptation across natural language tasks and domains, its emergence is less straightforward for modalities beyond text. In this work, we systematically uncover properties present in LLMs that support the emergence of ICL for autoregressive models and various modalities by promoting the learning of the needed mechanisms for ICL. We identify exact token repetitions in the training data sequences as an important factor for ICL. Such repetitions further improve stability and reduce transiency in ICL performance. Moreover, we emphasise the significance of training task difficulty for the emergence of ICL. Finally, by applying our novel insights on ICL emergence, we unlock ICL capabilities for various visual datasets and a more challenging EEG classification task.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 49 of its 72 references that are in the map. First posted on arXiv 9 Jan 2025.

Builds on

49 of 72 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  10. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  11. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  12. 2022In-context Learning and Induction Headscited by 930
37 more
  1. 2022Transformers learn in-context by gradient descentcited by 714
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  3. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  4. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  5. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  6. 2021Learning To Retrieve Prompts for In-Context Learningcited by 641
  7. 2021MetaICL: Learning to Learn In Contextcited by 450
  8. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  9. 2018Meta-learning with differentiable closed-form solverscited by 395
  10. 2024Large Brain Model for Learning Generic Representations with Tremendous EEG Data in BCIcited by 311
  11. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  12. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  13. 2022Visual Prompting via Image Inpaintingcited by 248
  14. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 240
  15. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  16. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  17. 2024Many-Shot In-Context Learningcited by 203
  18. 2023What Makes Good Examples for Visual In-Context Learning?cited by 150
  19. 2023The Learnability of In-Context Learningcited by 126
  20. 2023The mechanistic basis of data dependence and abrupt learning in an in-context classification taskcited by 115
  21. 2024The Evolution of Statistical Induction Heads: In-Context Learning Markov Chainscited by 115
  22. 2024In-Context Language Learning: Architectures and Algorithmscited by 86
  23. 2024What needs to go right for an induction head? A mechanistic study of in-context learning circuits and their formationcited by 86
  24. 2024Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvementscited by 82
  25. 2023Decoding Natural Images from EEG for Object Recognitioncited by 79
  26. 2023The Transient Nature of Emergent In-Context Learning in Transformerscited by 72
  27. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  28. 2023Pre-Training to Learn in Contextcited by 41
  29. 2023Understanding In-Context Learning via Supportive Pretraining Datacited by 41
  30. 2023BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decodingcited by 36
  31. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  32. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  33. 2023Auto-ICL: In-Context Learning without Human Supervisioncited by 18
  34. 2024Inferring Functionality of Attention Heads from their Parameterscited by 17
  35. 2023Adapt in Contexts: Retrieval-Augmented Domain Adaptation via In-Context Learningcited by 16
  36. 2024Visual Prompt Selection for In-Context Learning Segmentationcited by 12
  37. 2024InCoRo: In-Context Learning for Robotics Control with Feedback Loopscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Simulated Adoption: Decoupling Magnitude and Direction in LLM In-Context Conflict Resolutioncited by 0
  2. 2026Contrast Matters: Understanding Robustness of In-Context Fine-Tuning to Target-Context Relatednesscited by 0
  3. 2026Beyond Model Size: Probing the Gaps in Visual in-Context Learning by Training a Tiny Modelcited by 0
  4. 2026Quo Vadis, Visual In-Context Learning? A Unified Benchmark Across Domains and Taskscited by 0