# Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers

arXiv: https://arxiv.org/abs/2210.07362 · HTML: https://mlsanity.com/paper/2210.07362/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://mlsanity.com/paper/2004.09095/) (2020)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Unsupervised Domain Clusters in Pretrained Language Models](https://mlsanity.com/paper/2004.02105/) (2020)
- [Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies](https://mlsanity.com/paper/2108.12084/) (2021)
- [Sustainable Modular Debiasing of Language Models](https://mlsanity.com/paper/2109.03646/) (2021)
- [Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers](https://mlsanity.com/paper/2005.11787/) (2020)
- [Modeling Empathy and Distress in Reaction to News Stories](https://mlsanity.com/paper/1808.10399/) (2018)
- [Welcome to the Modern World of Pronouns: Identity-Inclusive Natural Language Processing beyond Gender](https://mlsanity.com/paper/2202.11923/) (2022)
- [Multi2WOZ: A Robust Multilingual Dataset and Conversational Pretraining for Task-Oriented Dialog](https://mlsanity.com/paper/2205.10400/) (2022)
- [DS-TOD: Efficient Domain Specialization for Task Oriented Dialog](https://mlsanity.com/paper/2110.08395/) (2021)
- [Compositional Demographic Word Embeddings](https://mlsanity.com/paper/2010.02986/) (2020)
- [Fair and Argumentative Language Modeling for Computational Argumentation](https://mlsanity.com/paper/2204.04026/) (2022)
- [EmpathBERT: A BERT-based Framework for Demographic-aware Empathy Prediction](https://mlsanity.com/paper/2102.00272/) (2021)
- [Geographic Adaptation of Pretrained Language Models](https://mlsanity.com/paper/2203.08565/) (2022)

## Built on by (most cited first, 10 of 10)

- [Social Bias Probing: Fairness Benchmarking for Language Models](https://mlsanity.com/paper/2311.09090/) (2023)
- [GRASP: A Disagreement Analysis Framework to Assess Group Associations in Perspectives](https://mlsanity.com/paper/2311.05074/) (2023)
- [D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation](https://mlsanity.com/paper/2404.10857/) (2024)
- [SocioProbe: What, When, and Where Language Models Learn about Sociodemographics](https://mlsanity.com/paper/2211.04281/) (2022)
- [TADA: Efficient Task-Agnostic Domain Adaptation for Transformers](https://mlsanity.com/paper/2305.12717/) (2023)
- [Wisdom of Instruction-Tuned Language Model Crowds. Exploring Model Label Variation](https://mlsanity.com/paper/2307.12973/) (2023)
- [Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?](https://mlsanity.com/paper/2401.12492/) (2024)
- [What the Weight?! A Unified Framework for Zero-Shot Knowledge Composition](https://mlsanity.com/paper/2401.12756/) (2024)
- [Transferring Fairness using Multi-Task Learning with Limited Demographic Information](https://mlsanity.com/paper/2305.12671/) (2023)
- [Phonotactic Complexity across Dialects](https://mlsanity.com/paper/2402.12998/) (2024)

## Abstract (arXiv)

Demographic factors (e.g., gender or age) shape our language. Previous work showed that incorporating demographic factors can consistently improve performance for various NLP tasks with traditional NLP models. In this work, we investigate whether these previous findings still hold with state-of-the-art pretrained Transformer-based language models (PLMs). We use three common specialization methods proven effective for incorporating external knowledge into pretrained Transformers (e.g., domain-specific or geographic knowledge). We adapt the language representations for the demographic dimensions of gender and age, using continuous language modeling and dynamic multi-task learning for adaptation, where we couple language modeling objectives with the prediction of demographic classes. Our results, when employing a multilingual PLM, show substantial gains in task performance across four languages (English, German, French, and Danish), which is consistent with the results of previous work. However, controlling for confounding factors - primarily domain and language proficiency of Transformer-based PLMs - shows that downstream performance gains from our demographic adaptation do not actually stem from demographic knowledge. Our results indicate that demographic specialization of PLMs, while holding promise for positive societal impact, still represents an unsolved problem for (modern) NLP.