# Characterizing English Variation across Social Media Communities with BERT

arXiv: https://arxiv.org/abs/2102.06820 · HTML: https://mlsanity.com/paper/2102.06820/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 85 references that are in the map.
- First posted on arXiv 12 Feb 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Analysing Lexical Semantic Change with Contextualised Word Representations](https://mlsanity.com/paper/2004.14118/) (2020)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [Computational Sociolinguistics: A Survey](https://mlsanity.com/paper/1508.07544/) (2015)
- [Overcoming Language Variation in Sentiment Analysis with Social Attention](https://mlsanity.com/paper/1511.06052/) (2015)
- [Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations](https://mlsanity.com/paper/1910.00194/) (2019)
- [Community Identity and User Engagement in a Multi-Community Landscape](https://mlsanity.com/paper/1705.09665/) (2017)
- [Word Sense Induction with Neural biLM and Symmetric Patterns](https://mlsanity.com/paper/1808.08518/) (2018)
- [Towards better substitution-based word sense induction](https://mlsanity.com/paper/1905.12598/) (2019)
- [Characterizing the Google Books corpus: Strong limits to inferences of socio-cultural and linguistic evolution](https://mlsanity.com/paper/1501.00960/) (2015)
- [Making "fetch" happen: The influence of social and linguistic context on nonstandard word growth and decline](https://mlsanity.com/paper/1802.04140/) (2018)
- [Divergent discourse between protests and counter-protests: #BlackLivesMatter and #AllLivesMatter](https://mlsanity.com/paper/1606.06820/) (2016)
- [Loyalty in Online Communities](https://mlsanity.com/paper/1703.03386/) (2017)
- [The Road to Success: Assessing the Fate of Linguistic Innovations in Online Communities](https://mlsanity.com/paper/1806.05838/) (2018)
- [Semantic Variation in Online Communities of Practice](https://mlsanity.com/paper/1806.05847/) (2018)
- [AutoSense Model for Word Sense Induction](https://mlsanity.com/paper/1811.09242/) (2018)
- [Sense Embedding Learning for Word Sense Induction](https://mlsanity.com/paper/1606.05409/) (2016)
- [#anorexia, #anarexia, #anarexyia: Characterizing Online Community Practices with Orthographic Variation](https://mlsanity.com/paper/1712.01411/) (2017)
- [Computing Classic Closeness Centrality, at Scale](https://mlsanity.com/paper/1409.0035/) (2014)
- [The Status Gradient of Trends in Social Media](https://mlsanity.com/paper/1603.03303/) (2016)

## Built on by (most cited first, 20 of 20)

- [Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models](https://mlsanity.com/paper/2208.03306/) (2022)
- [DEMix Layers: Disentangling Domains for Modular Language Modeling](https://mlsanity.com/paper/2108.05036/) (2021)
- [Scaling Expert Language Models with Unsupervised Domain Discovery](https://mlsanity.com/paper/2303.14177/) (2023)
- [Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection](https://mlsanity.com/paper/2201.10474/) (2022)
- [Culture is Not Trivia: Sociocultural Theory for Cultural NLP](https://mlsanity.com/paper/2502.12057/) (2025)
- [Words as Gatekeepers: Measuring Discipline-specific Terms and Meanings in Scholarly Publications](https://mlsanity.com/paper/2212.09676/) (2022)
- [DisCGen: A Framework for Discourse-Informed Counterspeech Generation](https://mlsanity.com/paper/2311.18147/) (2023)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [ValueScope: Unveiling Implicit Norms and Values via Return Potential Model of Social Interactions](https://mlsanity.com/paper/2407.02472/) (2024)
- [What company do words keep? Revisiting the distributional semantics of J.R. Firth & Zellig Harris](https://mlsanity.com/paper/2205.07750/) (2022)
- [Toward Informal Language Processing: Knowledge of Slang in Large Language Models](https://mlsanity.com/paper/2404.02323/) (2024)
- [Stability of Syntactic Dialect Classification Over Space and Time](https://mlsanity.com/paper/2209.04958/) (2022)
- [Social Meme-ing: Measuring Linguistic Variation in Memes](https://mlsanity.com/paper/2311.09130/) (2023)
- [Tracing Semantic Variation in Slang](https://mlsanity.com/paper/2210.08635/) (2022)
- [Inducing lexicons of in-group language with socio-temporal context](https://mlsanity.com/paper/2409.19257/) (2024)
- [Affect, Body, Cognition, Demographics, and Emotion: The ABCDE of Text Features for Computational Affective Science](https://mlsanity.com/paper/2512.17752/) (2025)
- [Syntactic Variation Across the Grammar: Modelling a Complex Adaptive System](https://mlsanity.com/paper/2309.11869/) (2023)
- [Sentiment Analysis in Digital Spaces: An Overview of Reviews](https://mlsanity.com/paper/2310.19687/) (2023)
- [Finding Common Ground: Graded Communal Knowledge in Bluesky Starter Packs](https://mlsanity.com/paper/2609.19549/) (2026)
- [From Utterances to Networks: Modelling Slang Adoption and Diffusion Across Subreddits](https://mlsanity.com/paper/2609.25669/) (2026)

## Abstract (arXiv)

Much previous work characterizing language variation across Internet social groups has focused on the types of words used by these groups. We extend this type of study by employing BERT to characterize variation in the senses of words as well, analyzing two months of English comments in 474 Reddit communities. The specificity of different sense clusters to a community, combined with the specificity of a community's unique word types, is used to identify cases where a social group's language deviates from the norm. We validate our metrics using user-created glossaries and draw on sociolinguistic theories to connect language variation with trends in community behavior. We find that communities with highly distinctive language are medium-sized, and their loyal and highly engaged users interact in dense networks.