# Automatic Domain Adaptation by Transformers in In-Context Learning

arXiv: https://arxiv.org/abs/2405.16819 · HTML: https://mlsanity.com/paper/2405.16819/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 30 references that are in the map.
- First posted on arXiv 27 May 2024.

## Builds on (most cited first, 15 of 15)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Domain Adaptation for Medical Image Analysis: A Survey](https://mlsanity.com/paper/2102.09508/) (2021)
- [Trained Transformers Learn Linear Models In-Context](https://mlsanity.com/paper/2306.09927/) (2023)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection](https://mlsanity.com/paper/2306.04637/) (2023)
- [Transformers learn to implement preconditioned gradient descent for in-context learning](https://mlsanity.com/paper/2306.00297/) (2023)
- [Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression](https://mlsanity.com/paper/2306.15063/) (2023)
- [Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes](https://mlsanity.com/paper/1906.07697/) (2019)
- [Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining](https://mlsanity.com/paper/2310.08566/) (2023)
- [A Short Survey on Importance Weighting for Machine Learning](https://mlsanity.com/paper/2403.10175/) (2024)

## Built on by (most cited first, 7 of 7)

- [Transformers Meet In-Context Learning: A Universal Approximation Theory](https://mlsanity.com/paper/2506.05200/) (2025)
- [Weight-Space Linear Recurrent Neural Networks](https://mlsanity.com/paper/2506.01153/) (2025)
- [Optimal In-context Adaptivity and Distributional Robustness of Transformers](https://mlsanity.com/paper/2510.23254/) (2025)
- [Beyond the Prompt in Large Language Models: Comprehension, In-Context Learning, and Chain-of-Thought](https://mlsanity.com/paper/2603.10000/) (2026)
- [An extension of linear self-attention for in-context learning](https://mlsanity.com/paper/2503.23814/) (2025)
- [Adaptive Weighted Loss for Sequential Recommendations on Sparse Domains](https://mlsanity.com/paper/2510.04375/) (2025)
- [A Unified Framework for In-Context Learning with Causal and Masked Language Models](https://mlsanity.com/paper/2607.04081/) (2026)

## Abstract (arXiv)

Selecting or designing an appropriate domain adaptation algorithm for a given problem remains challenging. This paper presents a Transformer model that can provably approximate and opt for domain adaptation methods for a given dataset in the in-context learning framework, where a foundation model performs new tasks without updating its parameters at test time. Specifically, we prove that Transformers can approximate instance-based and feature-based unsupervised domain adaptation algorithms and automatically select an algorithm suited for a given dataset. Numerical results indicate that in-context learning demonstrates an adaptive domain adaptation surpassing existing methods.