# DALLMi: Domain Adaption for LLM-based Multi-label Classifier

arXiv: https://arxiv.org/abs/2405.01883 · HTML: https://mlsanity.com/paper/2405.01883/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 3 May 2024.

## Builds on (most cited first, 15 of 15)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Unified Deep Supervised Domain Adaptation and Generalization](https://mlsanity.com/paper/1709.10190/) (2017)
- [Learning from positive and unlabeled data: a survey](https://mlsanity.com/paper/1811.04820/) (2018)
- [Text Classification via Large Language Models](https://mlsanity.com/paper/2305.08377/) (2023)
- [Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification](https://mlsanity.com/paper/1908.11860/) (2019)
- [Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration](https://mlsanity.com/paper/2107.05446/) (2021)
- [Efficient Domain Adaptation of Language Models via Adaptive Tokenization](https://mlsanity.com/paper/2109.07460/) (2021)
- [Efficient Hierarchical Domain Adaptation for Pretrained Language Models](https://mlsanity.com/paper/2112.08786/) (2021)
- [Knowledge Distillation for BERT Unsupervised Domain Adaptation](https://mlsanity.com/paper/2010.11478/) (2020)
- [The Trade-offs of Domain Adaptation for Neural Language Models](https://mlsanity.com/paper/2109.10274/) (2021)
- [Semi-supervised Models are Strong Unsupervised Domain Adaptation Learners](https://mlsanity.com/paper/2106.00417/) (2021)
- [Localising In-Domain Adaptation of Transformer-Based Biomedical Language Models](https://mlsanity.com/paper/2212.10422/) (2022)
- [BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweets](https://mlsanity.com/paper/2109.10234/) (2021)
- [Positive Label Is All You Need for Multi-Label Classification](https://mlsanity.com/paper/2306.16016/) (2023)
- [Discriminator-free Unsupervised Domain Adaptation for Multi-label Image Classification](https://mlsanity.com/paper/2301.10611/) (2023)

## Built on by (most cited first, 7 of 7)

- [Large Language Models Do Multi-Label Classification Differently](https://mlsanity.com/paper/2505.17510/) (2025)
- [Adaptable and Reliable Text Classification using Large Language Models](https://mlsanity.com/paper/2405.10523/) (2024)
- [On-Air Deep Learning Integrated Semantic Inference Models for Enhanced Earth Observation Satellite Networks](https://mlsanity.com/paper/2409.15246/) (2024)
- [Estimating Causal Effects of Text Interventions Leveraging LLMs](https://mlsanity.com/paper/2410.21474/) (2024)
- [Large Language Models for Healthcare Text Classification: A Systematic Review](https://mlsanity.com/paper/2503.01159/) (2025)
- [QUAD-LLM-MLTC: Large Language Models Ensemble Learning for Healthcare Text Multi-Label Classification](https://mlsanity.com/paper/2502.14189/) (2025)
- [Learning to vary: Teaching LMs to reproduce human linguistic variability in next-word prediction](https://mlsanity.com/paper/2509.17794/) (2025)

## Abstract (arXiv)

Large language models (LLMs) increasingly serve as the backbone for classifying text associated with distinct domains and simultaneously several labels (classes). When encountering domain shifts, e.g., classifier of movie reviews from IMDb to Rotten Tomatoes, adapting such an LLM-based multi-label classifier is challenging due to incomplete label sets at the target domain and daunting training overhead. The existing domain adaptation methods address either image multi-label classifiers or text binary classifiers. In this paper, we design DALLMi, Domain Adaptation Large Language Model interpolator, a first-of-its-kind semi-supervised domain adaptation method for text data models based on LLMs, specifically BERT. The core of DALLMi is the novel variation loss and MixUp regularization, which jointly leverage the limited positively labeled and large quantity of unlabeled text and, importantly, their interpolation from the BERT word embeddings. DALLMi also introduces a label-balanced sampling strategy to overcome the imbalance between labeled and unlabeled data. We evaluate DALLMi against the partial-supervised and unsupervised approach on three datasets under different scenarios of label availability for the target domain. Our results show that DALLMi achieves higher mAP than unsupervised and partially-supervised approaches by 19.9% and 52.2%, respectively.