# Similarity-Based Domain Adaptation with LLMs

arXiv: https://arxiv.org/abs/2503.05281 · HTML: https://mlsanity.com/paper/2503.05281/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 7 Mar 2025.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021)
- [Asymmetric Tri-training for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1702.08400/) (2017)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [Unsupervised Domain Adaptation through Self-Supervision](https://mlsanity.com/paper/1909.11825/) (2019)
- [Cross-Domain Gradient Discrepancy Minimization for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2106.04151/) (2021)
- [PERL: Pivot-based Domain Adaptation for Pre-trained Deep Contextualized Embedding Models](https://mlsanity.com/paper/2006.09075/) (2020)
- [UDALM: Unsupervised Domain Adaptation through Language Modeling](https://mlsanity.com/paper/2104.07078/) (2021)
- [Transformer Based Multi-Source Domain Adaptation](https://mlsanity.com/paper/2009.07806/) (2020)
- [Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations](https://mlsanity.com/paper/2301.02724/) (2023)
- [Knowledge Distillation for BERT Unsupervised Domain Adaptation](https://mlsanity.com/paper/2010.11478/) (2020)
- [Shallow Discourse Annotation for Chinese TED Talks](https://mlsanity.com/paper/2003.04032/) (2020)
- [Multi-Label Classification for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2406.04461/) (2024)
- [Domain-Invariant Feature Distillation for Cross-Domain Sentiment Classification](https://mlsanity.com/paper/1908.09122/) (2019)
- [Leveraging Hierarchical Prototypes as the Verbalizer for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2411.14880/) (2024)
- [On Evolving Attention Towards Domain Adaptation](https://mlsanity.com/paper/2103.13561/) (2021)
- [A Two-Stage Framework with Self-Supervised Distillation For Cross-Domain Text Classification](https://mlsanity.com/paper/2304.09820/) (2023)
- [Teacher Network Calibration Improves Cross-Quality Knowledge Distillation](https://mlsanity.com/paper/2304.07593/) (2023)

## Abstract (arXiv)

Unsupervised domain adaptation leverages abundant labeled data from various source domains to generalize onto unlabeled target data. Prior research has primarily focused on learning domain-invariant features across the source and target domains. However, these methods often require training a model using source domain data, which is time-consuming and can limit model usage for applications with different source data. This paper introduces a simple framework that utilizes the impressive generalization capabilities of Large Language Models (LLMs) for target data annotation without the need of source model training, followed by a novel similarity-based knowledge distillation loss. Our extensive experiments on cross-domain text classification reveal that our framework achieves impressive performance, specifically, 2.44% accuracy improvement when compared to the SOTA method.