# Can Humans Identify Domains?

arXiv: https://arxiv.org/abs/2404.01785 · HTML: https://mlsanity.com/paper/2404.01785/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 71 references that are in the map.
- First posted on arXiv 2 Apr 2024.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotations](https://mlsanity.com/paper/2110.05719/) (2021)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)
- [What Can We Learn from Collective Human Opinions on Natural Language Inference Data?](https://mlsanity.com/paper/2010.03532/) (2020)
- [Neural Unsupervised Domain Adaptation in NLP---A Survey](https://mlsanity.com/paper/2006.00632/) (2020)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [CrossNER: Evaluating Cross-Domain Named Entity Recognition](https://mlsanity.com/paper/2012.04373/) (2020)
- [Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning](https://mlsanity.com/paper/2009.10277/) (2020)
- [Learning to select data for transfer learning with Bayesian Optimization](https://mlsanity.com/paper/1707.05246/) (2017)
- [Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP](https://mlsanity.com/paper/2005.14672/) (2020)
- [Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey](https://mlsanity.com/paper/2104.06951/) (2021)
- [Whose Opinions Matter? Perspective-aware Models to Identify Opinions of Hate Speech Victims in Abusive Language Detection](https://mlsanity.com/paper/2106.15896/) (2021)
- [Out-of-Domain Detection for Low-Resource Text Classification Tasks](https://mlsanity.com/paper/1909.05357/) (2019)
- [What to do about non-standard (or non-canonical) language in NLP](https://mlsanity.com/paper/1608.07836/) (2016)
- [UDALM: Unsupervised Domain Adaptation through Language Modeling](https://mlsanity.com/paper/2104.07078/) (2021)
- [Domain Divergences: a Survey and Empirical Analysis](https://mlsanity.com/paper/2010.12198/) (2020)
- [CrossRE: A Cross-Domain Dataset for Relation Extraction](https://mlsanity.com/paper/2210.09345/) (2022)
- [Genre as Weak Supervision for Cross-lingual Dependency Parsing](https://mlsanity.com/paper/2109.04733/) (2021)

## Built on by (most cited first, 2 of 2)

- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [Beyond Dataset Creation: Critical View of Annotation Variation and Bias Probing of a Dataset for Online Radical Content Detection](https://mlsanity.com/paper/2412.11745/) (2024)

## Abstract (arXiv)

Textual domain is a crucial property within the Natural Language Processing (NLP) community due to its effects on downstream model performance. The concept itself is, however, loosely defined and, in practice, refers to any non-typological property, such as genre, topic, medium or style of a document. We investigate the core notion of domains via human proficiency in identifying related intrinsic textual properties, specifically the concepts of genre (communicative purpose) and topic (subject matter). We publish our annotations in *TGeGUM*: A collection of 9.1k sentences from the GUM dataset (Zeldes, 2017) with single sentence and larger context (i.e., prose) annotations for one of 11 genres (source type), and its topic/subtopic as per the Dewey Decimal library classification system (Dewey, 1979), consisting of 10/100 hierarchical topics of increased granularity. Each instance is annotated by three annotators, for a total of 32.7k annotations, allowing us to examine the level of human disagreement and the relative difficulty of each annotation task. With a Fleiss' kappa of at most 0.53 on the sentence level and 0.66 at the prose level, it is evident that despite the ubiquity of domains in NLP, there is little human consensus on how to define them. By training classifiers to perform the same task, we find that this uncertainty also extends to NLP models.