mlsanityData to 7 Oct 2026

Linear Representations of Sentiment in Large Language Models

arXiv 2310.15154 ↗First posted 23 Oct 2023cs.LG cs.AI cs.CLcited by 157

Not on the map. Why?

Abstract (arXiv)

Sentiment is a pervasive feature in natural language text, yet it is an open question how sentiment is represented within Large Language Models (LLMs). In this study, we reveal that across a range of models, sentiment is represented linearly: a single direction in activation space mostly captures the feature across a range of tasks with one extreme for positive and the other for negative. Through causal interventions, we isolate this direction and show it is causally relevant in both toy tasks and real world datasets such as Stanford Sentiment Treebank. Through this case study we model a thorough investigation of what a single direction means on a broad data distribution. We further uncover the mechanisms that involve this direction, highlighting the roles of a small subset of attention heads and neurons. Finally, we discover a phenomenon which we term the summarization motif: sentiment is not solely represented on emotionally charged words, but is additionally summarized at intermediate positions without inherent sentiment, such as punctuation and names. We show that in Stanford Sentiment Treebank zero-shot classification, 76% of above-chance classification accuracy is lost when ablating the sentiment direction, nearly half of which (36%) is due to ablating the summarized sentiment direction exclusively at comma positions.

Key facts. Cited by 157 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Oct 2023.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 48; 2025: 53; 2026: 52.

157 papers.

Sort:

By area:

  1. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  2. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  3. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  4. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  5. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  6. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  7. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  8. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  9. 2024Analyzing the Generalization and Reliability of Steering Vectorscited by 130
  10. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
  11. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  12. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
38 more
  1. 2025Model Organisms for Emergent Misalignmentcited by 84
  2. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  3. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  4. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  5. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  6. 2025Persona Features Control Emergent Misalignmentcited by 76
  7. 2025LLMs Encode Harmfulness and Refusal Separatelycited by 75
  8. 2025Convergent Linear Representations of Emergent Misalignmentcited by 66
  9. 2024Attention Heads of Large Language Models: A Surveycited by 65
  10. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  11. 2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
  12. 2024A Mechanistic Analysis of a Transformer Trained on a Symbolic Multi-Step Reasoning Taskcited by 49
  13. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  14. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  15. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 45
  16. 2025Angular Steering: Behavior Control via Rotation in Activation Spacecited by 45
  17. 2025Understanding (Un)Reliability of Steering Vectors in Language Modelscited by 39
  18. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  19. 2025FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steeringcited by 37
  20. 2024Multi-property Steering of Large Language Models with Dynamic Activation Compositioncited by 35
  21. 2025Beyond Semantics: Rediscovering Spatial Awareness in Vision-Language Modelscited by 35
  22. 2024Backward Lens: Projecting Language Model Gradients into the Vocabulary Spacecited by 30
  23. 2025Linear Representations of Political Perspective Emerge in Large Language Modelscited by 30
  24. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  25. 2024Representational Analysis of Binding in Language Modelscited by 27
  26. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  27. 2024Do LLMs "know" internally when they follow instructions?cited by 25
  28. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  29. 2024Beyond Single Concept Vector: Modeling Concept Subspace in LLMs with Gaussian Distributioncited by 24
  30. 2024Style-Specific Neurons for Steering LLMs in Text Style Transfercited by 24
  31. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  32. 2023Grokking Group Multiplication with Cosetscited by 22
  33. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  34. 2024Optimal ablation for interpretabilitycited by 21
  35. 2025EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identificationcited by 21
  36. 2025The Reasoning-Memorization Interplay in Language Models Is Mediated by a Single Directioncited by 21
  37. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  38. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19

50 most cited of 157 All 157 (JSON)