# Representing Affect Information in Word Embeddings

arXiv: https://arxiv.org/abs/2209.10583 · HTML: https://mlsanity.com/paper/2209.10583/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 42 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings](https://mlsanity.com/paper/1909.00512/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://mlsanity.com/paper/1907.13528/) (2019)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings](https://mlsanity.com/paper/1909.10430/) (2019)
- [Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words](https://mlsanity.com/paper/2005.01810/) (2020)
- [Contextualized Word Embeddings Encode Aspects of Human-Like Word Sense Knowledge](https://mlsanity.com/paper/2010.13057/) (2020)
- [Pragmatic competence of pre-trained language models through the lens of discourse connectives](https://mlsanity.com/paper/2109.12951/) (2021)
- [Using Knowledge-Embedded Attention to Augment Pre-trained Language Models for Fine-Grained Emotion Recognition](https://mlsanity.com/paper/2108.00194/) (2021)

## Built on by (most cited first, 3 of 3)

- [Emotions Where Art Thou: Understanding and Characterizing the Emotional Latent Space of Large Language Models](https://mlsanity.com/paper/2510.22042/) (2025)
- [Can Language Models Be Tricked by Language Illusions? Easier with Syntax, Harder with Semantics](https://mlsanity.com/paper/2311.01386/) (2023)
- [A Linguistic Comparison between Human and ChatGPT-Generated Conversations](https://mlsanity.com/paper/2401.16587/) (2024)

## Abstract (arXiv)

A growing body of research in natural language processing (NLP) and natural language understanding (NLU) is investigating human-like knowledge learned or encoded in the word embeddings from large language models. This is a step towards understanding what knowledge language models capture that resembles human understanding of language and communication. Here, we investigated whether and how the affect meaning of a word (i.e., valence, arousal, dominance) is encoded in word embeddings pre-trained in large neural networks. We used the human-labeled dataset as the ground truth and performed various correlational and classification tests on four types of word embeddings. The embeddings varied in being static or contextualized, and how much affect specific information was prioritized during the pre-training and fine-tuning phase. Our analyses show that word embedding from the vanilla BERT model did not saliently encode the affect information of English words. Only when the BERT model was fine-tuned on emotion-related tasks or contained extra contextualized information from emotion-rich contexts could the corresponding embedding encode more relevant affect information.