# KL Guided Domain Adaptation

arXiv: https://arxiv.org/abs/2106.07780 · HTML: https://mlsanity.com/paper/2106.07780/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 50 references that are in the map.
- First posted on arXiv 14 Jun 2021.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [In Search of Lost Domain Generalization](https://mlsanity.com/paper/2007.01434/) (2020)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [Contrastive Adaptation Network for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1901.00976/) (2019)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [NIPS 2016 Tutorial: Generative Adversarial Networks](https://mlsanity.com/paper/1701.00160/) (2016)
- [Domain Generalization for Object Recognition with Multi-task Autoencoders](https://mlsanity.com/paper/1508.07680/) (2015)
- [Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift](https://mlsanity.com/paper/2003.04475/) (2020)
- [Regularized Learning for Domain Adaptation under Label Shifts](https://mlsanity.com/paper/1903.09734/) (2019)
- [Support and Invertibility in Domain-Invariant Representations](https://mlsanity.com/paper/1903.03448/) (2019)
- [Domain-Adversarial Neural Networks](https://mlsanity.com/paper/1412.4446/) (2014)
- [Domain Invariant Representation Learning with Domain Density Transformations](https://mlsanity.com/paper/2102.05082/) (2021)
- [d-SNE: Domain Adaptation using Stochastic Neighborhood Embedding](https://mlsanity.com/paper/1905.12775/) (2019)
- [DIRL: Domain-Invariant Representation Learning for Sim-to-Real Transfer](https://mlsanity.com/paper/2011.07589/) (2020)

## Built on by (most cited first, 30 of 35)

- [Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models](https://mlsanity.com/paper/2404.02657/) (2024)
- [Optimal Representations for Covariate Shift](https://mlsanity.com/paper/2201.00057/) (2021)
- [CoCo: A Coupled Contrastive Framework for Unsupervised Domain Adaptive Graph Classification](https://mlsanity.com/paper/2306.04979/) (2023)
- [Perspective-Invariant 3D Object Detection](https://mlsanity.com/paper/2507.17665/) (2025)
- [Information-Theoretic Analysis of Unsupervised Domain Adaptation](https://mlsanity.com/paper/2210.00706/) (2022)
- [Wasserstein-regularized Conformal Prediction under General Distribution Shift](https://mlsanity.com/paper/2501.13430/) (2025)
- [Multimodal Misinformation Detection by Learning from Synthetic Data with Multimodal LLMs](https://mlsanity.com/paper/2409.19656/) (2024)
- [Gradual Domain Adaptation via Normalizing Flows](https://mlsanity.com/paper/2206.11492/) (2022)
- [On the Generalization for Transfer Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2207.05377/) (2022)
- [Test-time Adaptation for Regression by Subspace Alignment](https://mlsanity.com/paper/2410.03263/) (2024)
- [Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking](https://mlsanity.com/paper/2401.04266/) (2024)
- [Cauchy-Schwarz Divergence Information Bottleneck for Regression](https://mlsanity.com/paper/2404.17951/) (2024)
- [Federated Domain Generalization with Data-free On-server Matching Gradient](https://mlsanity.com/paper/2501.14653/) (2025)
- [The Conditional Cauchy-Schwarz Divergence with Applications to Time-Series Data and Sequential Decision Making](https://mlsanity.com/paper/2301.08970/) (2023)
- [On f-Divergence Principled Domain Adaptation: An Improved Framework](https://mlsanity.com/paper/2402.01887/) (2024)
- [Domain Adaptation with Cauchy-Schwarz Divergence](https://mlsanity.com/paper/2405.19978/) (2024)
- [Domain Adaptation and Entanglement: an Optimal Transport Perspective](https://mlsanity.com/paper/2503.08155/) (2025)
- [D2ADA: Dynamic Density-aware Active Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2202.06484/) (2022)
- [Non-stationary Domain Generalization: Theory and Algorithm](https://mlsanity.com/paper/2405.06816/) (2024)
- [IT-RUDA: Information Theory Assisted Robust Unsupervised Domain Adaptation](https://mlsanity.com/paper/2210.12947/) (2022)
- [How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysis](https://mlsanity.com/paper/2406.09745/) (2024)
- [DART: A Principled Approach to Adversarially Robust Unsupervised Domain Adaptation](https://mlsanity.com/paper/2402.11120/) (2024)
- [An Information-Theoretic Analysis for Federated Learning under Concept Drift](https://mlsanity.com/paper/2506.21036/) (2025)
- [Domain Generalization via Pareto Optimal Gradient Matching](https://mlsanity.com/paper/2507.14227/) (2025)
- [CoCoA-Mix: Confusion-and-Confidence-Aware Mixture Model for Context Optimization](https://mlsanity.com/paper/2506.07484/) (2025)
- [FedDAA: Dynamic Client Clustering for Concept Drift Adaptation in Federated Learning](https://mlsanity.com/paper/2506.21054/) (2025)
- [On the Hardness of Unsupervised Domain Adaptation: Optimal Learners and Information-Theoretic Perspective](https://mlsanity.com/paper/2507.06552/) (2025)
- [Open-Set Heterogeneous Domain Adaptation: Theoretical Analysis and Algorithm](https://mlsanity.com/paper/2412.13036/) (2024)
- [Stein Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2502.03587/) (2025)
- [Source-free domain adaptation based on label reliability for cross-domain bearing fault diagnosis](https://mlsanity.com/paper/2503.08749/) (2025)

## Abstract (arXiv)

Domain adaptation is an important problem and often needed for real-world applications. In this problem, instead of i.i.d. training and testing datapoints, we assume that the source (training) data and the target (testing) data have different distributions. With that setting, the empirical risk minimization training procedure often does not perform well, since it does not account for the change in the distribution. A common approach in the domain adaptation literature is to learn a representation of the input that has the same (marginal) distribution over the source and the target domain. However, these approaches often require additional networks and/or optimizing an adversarial (minimax) objective, which can be very expensive or unstable in practice. To improve upon these marginal alignment techniques, in this paper, we first derive a generalization bound for the target loss based on the training loss and the reverse Kullback-Leibler (KL) divergence between the source and the target representation distributions. Based on this bound, we derive an algorithm that minimizes the KL term to obtain a better generalization to the target domain. We show that with a probabilistic representation network, the KL term can be estimated efficiently via minibatch samples without any additional network or a minimax objective. This leads to a theoretically sound alignment method which is also very efficient and stable in practice. Experimental results also suggest that our method outperforms other representation-alignment approaches.