mlsanityData to 7 Oct 2026

Theoretical foundations and limits of word embeddings: what types of meaning can they capture?

arXiv 2107.10413 ↗First posted 22 Jul 2021cs.CL cs.CYcited by 5

Not on the map. Why?

Abstract (arXiv)

Measuring meaning is a central problem in cultural sociology and word embeddings may offer powerful new tools to do so. But like any tool, they build on and exert theoretical assumptions. In this paper I theorize the ways in which word embeddings model three core premises of a structural linguistic theory of meaning: that meaning is relational, coherent, and may be analyzed as a static system. In certain ways, word embedding methods are vulnerable to the same, enduring critiques of these premises. In other ways, they offer novel solutions to these critiques. More broadly, formalizing the study of meaning with word embeddings offers theoretical opportunities to clarify core concepts and debates in cultural sociology, such as the coherence of meaning. Just as network analysis specified the once vague notion of social relations (Borgatti et al. 2009), formalizing meaning with embedding methods can push us to specify and reimagine meaning itself.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 44 of its 155 references that are in the map. First posted on arXiv 22 Jul 2021.

Builds on

44 of 155 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  7. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,304
  8. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  9. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  10. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  11. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
32 more
  1. 2017Word Embeddings Quantify 100 Years of Gender and Ethnic Stereotypescited by 251
  2. 2020Experience Grounds Languagecited by 241
  3. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 219
  4. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  5. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  6. 2019Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overviewcited by 137
  7. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 113
  8. 2018Diachronic word embeddings and semantic shifts: a surveycited by 90
  9. 2018The Geometry of Culture: Analyzing Meaning through Word Embeddingscited by 87
  10. 2017Semi-supervised sequence tagging with bidirectional language modelscited by 86
  11. 2014Statistically Significant Detection of Linguistic Changecited by 82
  12. 2019Understanding Undesirable Word Embedding Associationscited by 59
  13. 2018Towards Understanding Linear Word Analogiescited by 57
  14. 2014Temporal Analysis of Language through Neural Language Modelscited by 53
  15. 2014word2vec Parameter Learning Explainedcited by 53
  16. 2017Dynamic Word Embeddingscited by 51
  17. 2017Dynamic Word Embeddings for Evolving Semantic Discoverycited by 38
  18. 2015Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Spacecited by 35
  19. 2016Problems With Evaluation of Word Embeddings Using Word Similarity Taskscited by 25
  20. 2017Are distributional representations ready for the real world? Evaluating word vectors for grounded perceptual meaningcited by 19
  21. 2020Contextual Embeddings: When Are They Worth It?cited by 15
  22. 2020When do Word Embeddings Accurately Reflect Surveys on our Beliefs About People?cited by 11
  23. 2018From Context to Concept: Exploring Semantic Relationships in Music with Word2Veccited by 9
  24. 2021Integrating topic modeling and word embedding to characterize violent deathscited by 8
  25. 2020Machine learning as a model for cultural learning: Teaching an algorithm what it means to be fatcited by 7
  26. 2020Cultural Cartography with Word Embeddingscited by 7
  27. 2017Evaluating vector-space models of analogycited by 6
  28. 2018Semantic projection: recovering human knowledge of multiple, distinct object features from word embeddingscited by 6
  29. 2017Sound-Word2Vec: Learning Word Representations Grounded in Soundscited by 4
  30. 2020Social Centralization and Semantic Collapse: Hyperbolic Embeddings of Networks and Textcited by 4
  31. 2017A World of Difference: Divergent Word Interpretations among Peoplecited by 1
  32. 2019Distributed Vector Representations of Folksong Motifscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 2; 2026: 2.

5 papers.

  1. 2025Navigating the Risks of Using Large Language Models for Text Annotation in Social Science Researchcited by 4
  2. 2025Computational Basis of LLM's Decision Making in Social Simulationcited by 2
  3. 2026The Prediction-Measurement Gap: Toward Meaning Representations as Scientific Instrumentscited by 1
  4. 2022DeepProphet2 -- A Deep Learning Gene Recommendation Enginecited by 0
  5. 2026Simulating Meaning, Nevermore! Introducing ICR: A Semiotic-Hermeneutic Metric for Evaluating Meaning in LLM Text Summariescited by 0