mlsanityData to 14 Oct 2026

Retrieval-Based In-Context Learning: A Domain Adaptation Framework

arXiv 2610.05717 ↗First posted 5 Oct 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

In-context retrieval (ICR) is a retrieval-based form of in-context learning (ICL) in which demonstrations are retrieved from a source database based on similarity to the query, rather than sampled independently. In this work, we formulate ICR as a type of domain adaptation problem, where the source distribution P of the database may differ from the target distribution Q of the test query-label pair. We investigate the performance of ICR under a flexible class of distributional shifts that substantially extends prior work citepli2024fine,guo2025retrieval, and establish theoretical guarantees that quantify the benefits and pitfalls of this learning paradigm. Our theory is verified by experiments on synthetic and language tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 5 Oct 2026.

Builds on

47 of 59 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  4. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  5. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  6. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  8. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  9. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  11. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  12. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
35 more
  1. 2021Learning To Retrieve Prompts for In-Context Learningcited by 648
  2. 2023Trained Transformers Learn Linear Models In-Contextcited by 337
  3. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
  4. 2025TabICL: A Tabular Foundation Model for In-Context Learning on Large Datacited by 289
  5. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  6. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  7. 2024Many-Shot In-Context Learningcited by 206
  8. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 151
  9. 2023Compositional Exemplars for In-context Learningcited by 144
  10. 2023Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sourcescited by 122
  11. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 113
  12. 2023Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrievalcited by 78
  13. 2023UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluationcited by 75
  14. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
  15. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 70
  16. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directionscited by 70
  17. 2019Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifiercited by 65
  18. 2020On the Value of Target Data in Transfer Learningcited by 65
  19. 2014Rates of Convergence for Nearest Neighbor Classificationcited by 46
  20. 2023MoT: Memory-of-Thought Enables ChatGPT to Self-Improvecited by 38
  21. 2024In-Context Learning with Representations: Contextual Generalization of Trained Transformerscited by 38
  22. 2023RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Modelscited by 37
  23. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 33
  24. 2024Fine-grained Analysis of In-context Linear Estimation: Data, Architecture, and Beyondcited by 24
  25. 2018A Generalized Neyman-Pearson Criterion for Optimal Domain Adaptationcited by 23
  26. 2021A linear adjustment based approach to posterior drift in transfer learningcited by 17
  27. 2024One-Layer Transformer Provably Learns One-Nearest Neighbor In Contextcited by 16
  28. 2025Test-Time Training Provably Improves Transformers as In-context Learnerscited by 16
  29. 2024Exploring the Robustness of In-Context Learning with Noisy Labelscited by 14
  30. 2024Label Noise: Ignorance Is Blisscited by 9
  31. 2025Empowering LLM Agents with Geospatial Awareness: Toward Grounded Reasoning for Wildfire Responsecited by 9
  32. 2025Out-of-Distribution Generalization of In-Context Learning: A Low-Dimensional Subspace Perspectivecited by 6
  33. 2023Robust Transfer Learning with Unreliable Source Datacited by 5
  34. 2025Retrieval-Augmented Generation as Noisy In-Context Learning: A Unified Theory and Risk Boundscited by 4
  35. 2025PAC Learnability in the Presence of Performativitycited by 1

Built on by

Not cited yet.