mlsanityData to 14 Oct 2026

Can Humans Identify Domains?

arXiv 2404.01785 ↗First posted 2 Apr 2024cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Textual domain is a crucial property within the Natural Language Processing (NLP) community due to its effects on downstream model performance. The concept itself is, however, loosely defined and, in practice, refers to any non-typological property, such as genre, topic, medium or style of a document. We investigate the core notion of domains via human proficiency in identifying related intrinsic textual properties, specifically the concepts of genre (communicative purpose) and topic (subject matter). We publish our annotations in *TGeGUM*: A collection of 9.1k sentences from the GUM dataset (Zeldes, 2017) with single sentence and larger context (i.e., prose) annotations for one of 11 genres (source type), and its topic/subtopic as per the Dewey Decimal library classification system (Dewey, 1979), consisting of 10/100 hierarchical topics of increased granularity. Each instance is annotated by three annotators, for a total of 32.7k annotations, allowing us to examine the level of human disagreement and the relative difficulty of each annotation task. With a Fleiss' kappa of at most 0.53 on the sentence level and 0.66 at the prose level, it is evident that despite the ubiquity of domains in NLP, there is little human consensus on how to define them. By training classifiers to perform the same task, we find that this uncertainty also extends to NLP models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 34 of its 71 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

34 of 71 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  9. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  10. 2023BloombergGPT: A Large Language Model for Financecited by 939
  11. 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 486
  12. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
22 more
  1. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  2. 2019Abductive Commonsense Reasoningcited by 301
  3. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  4. 2020What Can We Learn from Collective Human Opinions on Natural Language Inference Data?cited by 153
  5. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 144
  6. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
  7. 2020CrossNER: Evaluating Cross-Domain Named Entity Recognitioncited by 104
  8. 2020Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learningcited by 68
  9. 2017Learning to select data for transfer learning with Bayesian Optimizationcited by 64
  10. 2020Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLPcited by 50
  11. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  12. 2021Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detectioncited by 38
  13. 2019Out-of-Domain Detection for Low-Resource Text Classification Taskscited by 36
  14. 2016What to do about non-standard (or non-canonical) language in NLPcited by 35
  15. 2021UDALM: Unsupervised Domain Adaptation through Language Modelingcited by 33
  16. 2020Domain Divergences: a Survey and Empirical Analysiscited by 24
  17. 2022CrossRE: A Cross-Domain Dataset for Relation Extractioncited by 21
  18. 2021Genre as Weak Supervision for Cross-lingual Dependency Parsingcited by 8
  19. 2018Crowdsourcing Semantic Label Propagation in Relation Classificationcited by 6
  20. 2021On the Effectiveness of Dataset Embeddings in Mono-lingual,Multi-lingual and Zero-shot Conditionscited by 4
  21. 2021How Universal is Genre in Universal Dependencies?cited by 4
  22. 2023Estimating Semantic Similarity between In-Domain and Out-of-Domain Samplescited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  2. 2024Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detectioncited by 3