mlsanityData to 7 Oct 2026

Automatic Domain Adaptation by Transformers in In-Context Learning

arXiv 2405.16819 ↗First posted 27 May 2024cs.LG stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Selecting or designing an appropriate domain adaptation algorithm for a given problem remains challenging. This paper presents a Transformer model that can provably approximate and opt for domain adaptation methods for a given dataset in the in-context learning framework, where a foundation model performs new tasks without updating its parameters at test time. Specifically, we prove that Transformers can approximate instance-based and feature-based unsupervised domain adaptation algorithms and automatically select an algorithm suited for a given dataset. Numerical results indicate that in-context learning demonstrates an adaptive domain adaptation surpassing existing methods.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 15 of its 30 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

15 of 30 references.

References · 15

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  4. 2022Transformers learn in-context by gradient descentcited by 714
  5. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  6. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  7. 2021Domain Adaptation for Medical Image Analysis: A Surveycited by 351
  8. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  9. 2018Meta-Transfer Learning for Few-Shot Learningcited by 306
  10. 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
  12. 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 138
3 more
  1. 2019Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processescited by 131
  2. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 82
  3. 2024A Short Survey on Importance Weighting for Machine Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Weight-Space Linear Recurrent Neural Networkscited by 5
  2. 2025Transformers Meet In-Context Learning: A Universal Approximation Theorycited by 4
  3. 2025Optimal In-context Adaptivity and Distributional Robustness of Transformerscited by 1
  4. 2026Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thoughtcited by 1
  5. 2025An extension of linear self-attention for in-context learningcited by 0
  6. 2025Adaptive Weighted Loss for Sequential Recommendations on Sparse Domainscited by 0
  7. 2026A Unified Framework for In-Context Learning with Causal and Masked Language Modelscited by 0