mlsanityData to 7 Oct 2026

Unsupervised Domain Clusters in Pretrained Language Models

arXiv 2004.02105 ↗First posted 5 Apr 2020cs.CLcited by 160

Not on the map. Why?

Abstract (arXiv)

The notion of "in-domain data" in NLP is often over-simplistic and vague, as textual data varies in many nuanced linguistic aspects such as topic, style or level of formality. In addition, domain labels are many times unavailable, making it challenging to build domain-specific systems. We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision -- suggesting a simple data-driven definition of domains in textual data. We harness this property and propose domain data selection methods based on such models, which require only a small set of in-domain monolingual data. We evaluate our data selection methods for neural machine translation across five diverse domains, where they outperform an established approach as measured by both BLEU and by precision and recall of sentence selection with respect to an oracle.

Key facts. Cited by 160 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 32; 2022: 36; 2023: 42; 2024: 31; 2025: 14; 2026: 5.

160 papers.

Sort:

By area:

  1. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  2. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  3. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  4. 2022In-context Examples Selection for Machine Translationcited by 172
  5. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
  6. 2021An Interpretability Illusion for BERTcited by 94
  7. 2021Efficient Nearest Neighbor Language Modelscited by 93
  8. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  9. 2023Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translationcited by 74
  10. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  11. 2021Neural Machine Translation with Monolingual Translation Memorycited by 61
  12. 2023Goal Driven Discovery of Distributional Differences via Language Descriptionscited by 61
38 more
  1. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  2. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  3. 2023Goal-Driven Explainable Clustering via Language Descriptionscited by 58
  4. 2023Guiding Language Model Reasoning with Planning Tokenscited by 58
  5. 2022Describing Differences between Text Distributions with Natural Languagecited by 54
  6. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  7. 2023Steering Large Language Models for Machine Translation with Finetuning and In-Context Learningcited by 48
  8. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  9. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 46
  10. 2021Fast Nearest Neighbor Machine Translationcited by 37
  11. 2022T-NER: An All-Round Python Library for Transformer-based Named Entity Recognitioncited by 37
  12. 2022Is Neural Topic Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings for Topicscited by 35
  13. 2023Efficient Continual Pre-training for Building Domain Specific Large Language Modelscited by 33
  14. 2021Efficient Hierarchical Domain Adaptation for Pretrained Language Modelscited by 32
  15. 2024Get more for less: Principled Data Selection for Warming Up Fine-Tuning in LLMscited by 32
  16. 2022AdaPrompt: Adaptive Model Training for Prompt-based NLPcited by 30
  17. 2023Is Fine-tuning Needed? Pre-trained Language Models Are Near Perfect for Out-of-Domain Detectioncited by 29
  18. 2024TSDS: Data Selection for Task-Specific Model Finetuningcited by 28
  19. 2021GAIA: A Transfer Learning System of Object Detection that Fits Your Needscited by 27
  20. 2023CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Modelscited by 26
  21. 2021DirectProbe: Studying Representations without Classifierscited by 25
  22. 2021Learning Kernel-Smoothed Machine Translation with Retrieved Examplescited by 25
  23. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  24. 2021Are Pretrained Transformers Robust in Intent Classification? A Missing Ingredient in Evaluation of Out-of-Scope Intent Detectioncited by 22
  25. 2024The Fine-Tuning Paradox: Boosting Translation Quality Without Sacrificing LLM Abilitiescited by 22
  26. 2021DS-TOD: Efficient Domain Specialization for Task Oriented Dialogcited by 21
  27. 2021Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapterscited by 21
  28. 2022Chunk-based Nearest Neighbor Machine Translationcited by 21
  29. 2023Augmenting Large Language Model Translators via Translation Memoriescited by 21
  30. 2024Explaining Datasets in Words: Statistical Models with Natural Language Parameterscited by 21
  31. 2021(Un)solving Morphological Inflection: Lemma Overlap Artificially Inflates Models' Performancecited by 20
  32. 2021Non-Parametric Unsupervised Domain Adaptation for Neural Machine Translationcited by 20
  33. 2023There's no Data Like Better Data: Using QE Metrics for MT Data Filteringcited by 19
  34. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  35. 2021CiteWorth: Cite-Worthiness Detection for Improved Scientific Document Understandingcited by 17
  36. 2024Salute the Classic: Revisiting Challenges of Machine Translation in the Age of Large Language Modelscited by 17
  37. 2021Sequential Reptile: Inter-Task Gradient Alignment for Multilingual Learningcited by 16
  38. 2021Bandits Don't Follow Rules: Balancing Multi-Facet Machine Translation with Multi-Armed Banditscited by 16

50 most cited of 160 All 160 (JSON)