mlsanityData to 14 Oct 2026

Compositional Zero-Shot Domain Transfer with Text-to-Text Models

arXiv 2303.13386 ↗First posted 23 Mar 2023cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Label scarcity is a bottleneck for improving task performance in specialised domains. We propose a novel compositional transfer learning framework (DoT5 - domain compositional zero-shot T5) for zero-shot domain transfer. Without access to in-domain labels, DoT5 jointly learns domain knowledge (from MLM of unlabelled in-domain free text) and task knowledge (from task training on more readily available general-domain data) in a multi-task manner. To improve the transferability of task training, we design a strategy named NLGU: we simultaneously train NLG for in-domain label-to-data generation which enables data augmentation for self-finetuning and NLU for label prediction. We evaluate DoT5 on the biomedical domain and the resource-lean subdomain of radiology, focusing on NLI, text summarisation and embedding learning. DoT5 demonstrates the effectiveness of compositional transfer learning through multi-task learning. In particular, DoT5 outperforms the current SOTA in zero-shot transfer by over 7 absolute points in accuracy on RadNLI. We validate DoT5 with ablations and a case study demonstrating its ability to solve challenging NLI examples requiring in-domain expertise.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 37 of its 48 references that are in the map. First posted on arXiv 23 Mar 2023.

Builds on

37 of 48 references.

References · 37

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  8. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  9. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
25 more
  1. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  2. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  4. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  5. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  6. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 507
  7. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 478
  8. 2021MetaICL: Learning to Learn In Contextcited by 454
  9. 2022Making the Most of Text Semantics to Improve Biomedical Vision--Language Processingcited by 339
  10. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 337
  11. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 299
  12. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 207
  13. 2016Enhanced LSTM for Natural Language Inferencecited by 200
  14. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  15. 2018Lessons from Natural Language Inference in the Clinical Domaincited by 151
  16. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  17. 2020Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generationcited by 141
  18. 2021GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrievalcited by 119
  19. 2021SciFive: a text-to-text transformer model for biomedical literaturecited by 99
  20. 2018MedSTS: A Resource for Clinical Semantic Textual Similaritycited by 39
  21. 2018An Analysis of Attention Mechanisms: The Case of Word Sense Disambiguation in Neural Machine Translationcited by 15
  22. 2022Domain-Oriented Prefix-Tuning: Towards Efficient and Generalizable Fine-tuning for Zero-Shot Dialogue Summarizationcited by 14
  23. 2022Counterfactual Data Augmentation improves Factuality of Abstractive Summarizationcited by 8
  24. 2022Entity-driven Fact-aware Abstractive Summarization of Biomedical Literaturecited by 3
  25. 2022Task Transfer and Domain Adaptation for Zero-Shot Question Answeringcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 4; 2025: 1.

6 papers.

  1. 2024Instruction Tuning With Loss Over Instructionscited by 40
  2. 2023Exploring the Boundaries of GPT-4 in Radiologycited by 31
  3. 2024Unveiling LLM Evaluation Focused on Metrics: Challenges and Solutionscited by 18
  4. 2025From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corporacited by 7
  5. 2024Extracting and Encoding: Leveraging Large Language Models and Medical Knowledge to Enhance Radiological Text Representationcited by 5
  6. 2024AI for Biomedicine in the Era of Large Language Modelscited by 3