mlsanityData to 14 Oct 2026

Vector-ICL: In-context Learning with Continuous Vector Representations

arXiv 2410.05629 ↗First posted 8 Oct 2024cs.CL cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have shown remarkable in-context learning (ICL) capabilities on textual data. We explore whether these capabilities can be extended to continuous vectors from diverse domains, obtained from black-box pretrained encoders. By aligning input data with an LLM's embedding space through lightweight projectors, we observe that LLMs can effectively process and learn from these projected vectors, which we term Vector-ICL. In particular, we find that pretraining projectors with general language modeling objectives enables Vector-ICL, while task-specific finetuning further enhances performance. In our experiments across various tasks and modalities, including text reconstruction, numerical function regression, text classification, summarization, molecule captioning, time-series classification, graph classification, and fMRI decoding, Vector-ICL often surpasses both few-shot ICL and domain-specific model or tuning. We further conduct analyses and case studies, indicating the potential of LLMs to process vector representations beyond traditional token-based paradigms.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

48 of 62 references.

References · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  5. 2023Mistral 7Bcited by 3,164
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  11. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
36 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  2. 2024Chronos: Learning the Language of Time Seriescited by 759
  3. 2022Transformers learn in-context by gradient descentcited by 732
  4. 2024Yi: Open Foundation Models by 01.AIcited by 719
  5. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  6. 2021Large Dual Encoders Are Generalizable Retrieverscited by 456
  7. 2021MetaICL: Learning to Learn In Contextcited by 454
  8. 2018The UCR Time Series Archivecited by 431
  9. 2023Larger language models do in-context learning differentlycited by 394
  10. 2021Transformers Can Do Bayesian Inferencecited by 360
  11. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  12. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  13. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
  14. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  15. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  16. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  17. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  18. 2022Can language models learn from explanations in context?cited by 249
  19. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  20. 2024Many-Shot In-Context Learningcited by 206
  21. 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
  22. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  23. 2022Transformer Neural Processes: Uncertainty-Aware Meta Learning Via Sequence Modelingcited by 127
  24. 2023The Learnability of In-Context Learningcited by 127
  25. 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 96
  26. 2024Many-Shot In-Context Learning in Multimodal Foundation Modelscited by 67
  27. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 63
  28. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 63
  29. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 54
  30. 2023In-Context Learning with Many Demonstration Examplescited by 39
  31. 2024Parallel Structures in Pre-training Data Yield In-Context Learningcited by 26
  32. 2024How Well Can Transformers Emulate In-context Newton's Method?cited by 25
  33. 2024L+M-24: Building a Dataset for Language + Molecules @ ACL 2024cited by 21
  34. 2023Tree Prompting: Efficient Task Adaptation without Fine-Tuningcited by 18
  35. 2024Probing the Decision Boundaries of In-context Learning in Large Language Modelscited by 17
  36. 2024Crafting Interpretable Embeddings by Asking LLMs Questionscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 7; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025Hybrid Latent Reasoning via Reinforcement Learningcited by 32
  2. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 28
  3. 2025Representation Engineering for Large-Language Models: Survey and Research Challengescited by 21
  4. 2025Text Generation Beyond Discrete Token Samplingcited by 21
  5. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  6. 2024KBLaM: Knowledge Base augmented Language Modelcited by 16
  7. 2025OMNIGUARD: An Efficient Approach for AI Safety Moderation Across Languages and Modalitiescited by 7
  8. 2025Supervised In-Context Fine-Tuning for Generative Sequence Labelingcited by 1
  9. 2026Parallel In-context Learning for Large Vision Language Modelscited by 1
  10. 2025Next-Token Prediction Should be Ambiguity-Sensitive: A Meta-Learning Perspectivecited by 0
  11. 2026Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoderscited by 0
  12. 2026Data-Efficient Adaptation of LLMs via Attention Head Reweightingcited by 0
1 more
  1. 2026FETERS: Few-Shot Early Time-Series Classification via Effective Ratio Selectioncited by 0