mlsanityData to 14 Oct 2026

Implicit In-context Learning

arXiv 2405.14660 ↗First posted 23 May 2024cs.LG cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

In-context Learning (ICL) empowers large language models (LLMs) to swiftly adapt to unseen tasks at inference-time by prefixing a few demonstration examples before queries. Despite its versatility, ICL incurs substantial computational and memory overheads compared to zero-shot learning and is sensitive to the selection and order of demonstration examples. In this work, we introduce Implicit In-context Learning (I2CL), an innovative paradigm that reduces the inference cost of ICL to that of zero-shot learning with minimal information loss. I2CL operates by first generating a condensed vector representation, namely a context vector, extracted from the demonstration examples. It then conducts an inference-time intervention through injecting a linear combination of the context vector and query activations back into the model's residual streams. Empirical evaluation on nine real-world tasks across three model architectures demonstrates that I2CL achieves few-shot level performance at zero-shot inference cost, and it exhibits robustness against variations in demonstration examples. Furthermore, I2CL facilitates a novel representation of task-ids, enhancing task similarity detection and fostering effective transfer learning. We also perform a comprehensive analysis and ablation study on I2CL, offering deeper insights into its internal mechanisms. Code is available at https://github.com/LzVv123456/I2CL.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 36 of its 58 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

36 of 58 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  5. 2022Emergent Abilities of Large Language Modelscited by 2,512
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  11. 2019A Comprehensive Survey on Transfer Learningcited by 1,160
  12. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
24 more
  1. 2022Mass-Editing Memory in a Transformercited by 1,001
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  4. 2022Transformers learn in-context by gradient descentcited by 732
  5. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  7. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  8. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  9. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  10. 2023Adapting Language Models to Compress Contextscited by 356
  11. 2023In-Context Learning Creates Task Vectorscited by 312
  12. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  13. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  14. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 200
  15. 2023In-context Autoencoder for Context Compression in a Large Language Modelcited by 199
  16. 2018Hate Speech Dataset from a White Supremacy Forumcited by 198
  17. 2021Implicit Representations of Meaning in Neural Language Modelscited by 180
  18. 2021Noisy Channel Language Model Prompting for Few-Shot Text Classificationcited by 165
  19. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 161
  20. 2022An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labelscited by 89
  21. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  22. 2022Structured Prompting: Scaling In-Context Learning to 1,000 Examplescited by 81
  23. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  24. 2022Robustness of Demonstration-based Learning Under Limited Data Scenariocited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 66
  2. 2025M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning via Representation Engineeringcited by 35
  3. 2025Just-in-time and distributed task representations in language modelscited by 9
  4. 2025Train Once, Reuse Everywhere: Generalizable Implicit In-Context Learning by Routing Attentioncited by 8
  5. 2025Adaptive Task Vectors for Large Language Modelscited by 5
  6. 2025Transmuting prompts into weightscited by 3
  7. 2026Context Learning for Multi-Agent Discussioncited by 3
  8. 2026Enhancing Multimodal In-Context Learning via Inductive-Deductive Reasoningcited by 2
  9. 2025Examples as the Prompt: A Scalable Approach for Efficient LLM Adaptation in E-Commercecited by 1
  10. 2025Large Sign Language Models: Toward 3D American Sign Language Translationcited by 1
  11. 2026M2G-LLM: Enhancing Clinical Prediction via Multimodal Graph Reasoning and LLM Context Injectioncited by 0