mlsanityData to 7 Oct 2026

The Sociolinguistic Foundations of Language Modeling

arXiv 2407.09241 ↗First posted 12 Jul 2024cs.CLcited by 13

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce a sociolinguistic perspective on language modeling. We claim that large language models are inherently models of varieties of language, and we consider how this insight can inform the development and deployment of large language models. We begin by presenting a technical definition of the concept of a variety of language as developed in sociolinguistics. We then discuss how this perspective can help address five basic challenges in language modeling: social bias, domain adaptation, alignment, language change, and scale. Ultimately, we argue that it is crucial to carefully define and compile training corpora that accurately represent the specific varieties of language being modeled to maximize the performance and societal value of large language models.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 62 of its 166 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

62 of 166 references.

References · 62

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2023Mistral 7Bcited by 3,065
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
38 more
  1. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  2. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  5. 2020Aligning AI With Shared Human Valuescited by 706
  6. 2023A Watermark for Large Language Modelscited by 667
  7. 2020Artificial Intelligence, Values and Alignmentcited by 443
  8. 2021Explaining Neural Scaling Lawscited by 353
  9. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  10. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  11. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  12. 2023Large Language Model Alignment: A Surveycited by 241
  13. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  14. 2019Identifying and Reducing Gender Bias in Word-Level Language Modelscited by 214
  15. 2016Demographic Dialectal Variation in Social Media: A Case Study of African-American Englishcited by 194
  16. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  17. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  18. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 155
  19. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  20. 2023Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Modelscited by 144
  21. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  22. 2023Large language models in healthcare and medical domain: A reviewcited by 126
  23. 2021Truthful AI: Developing and governing AI that does not liecited by 109
  24. 2021Detoxifying Language Models Risks Marginalizing Minority Voicescited by 106
  25. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 98
  26. 2023Do We Still Need Clinical Language Models?cited by 97
  27. 2021On Measures of Biases and Harms in NLPcited by 81
  28. 2017Racial Disparity in Natural Language Processing: A Case Study of Social Media African-American Englishcited by 69
  29. 2023ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishingcited by 51
  30. 2023Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Votingcited by 46
  31. 2015Computational Sociolinguistics: A Surveycited by 35
  32. 2024Large Language Models Are Neurosymbolic Reasonerscited by 31
  33. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  34. 2024LoTR: Low Tensor Rank Weight Adaptationcited by 27
  35. 2024From Representational Harms to Quality-of-Service Harms: A Case Study on Llama 2 Safety Safeguardscited by 17
  36. 2024A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industrycited by 17
  37. 2023Fairness in Language Models Beyond English: Gaps and Challengescited by 16
  38. 2018Making "fetch" happen: The influence of social and linguistic context on nonstandard word growth and declinecited by 15

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 10.

13 papers.

Sort:

  1. 2025Persona-Augmented Benchmarking: Evaluating LLMs Across Diverse Writing Stylescited by 15
  2. 2025LLM-Based Social Simulations Require a Boundarycited by 11
  3. 2025When Do Large Language Models Exhibit Unsolicited Deception?cited by 9
  4. 2025Meta-Cultural Competence: Climbing the Right Hill of Cultural Awarenesscited by 8
  5. 2025Automatic Speech Recognition Biases in Newcastle English: an Error Analysiscited by 6
  6. 2024Large corpora and large language models: a replicable method for automating grammatical annotationcited by 5
  7. 2025Toward a Theory of Generalizability in LLM Mechanistic Interpretability Researchcited by 4
  8. 2024Performance in a dialectal profiling task of LLMs for varieties of Brazilian Portuguesecited by 3
  9. 2025We Need to Measure Data Diversity in NLP -- Better and Broadercited by 2
  10. 2025References Matter: Investigating the Impact of Reference Set Variation on Summarization Evaluationcited by 2
  11. 2024Dialetto, ma Quanto Dialetto? Transcribing and Evaluating Dialects on a Continuumcited by 1
  12. 2025ChatGPT-generated texts show authorship traits that identify them as non-humancited by 1
1 more
  1. 2025Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approachcited by 0