mlsanityData to 14 Oct 2026

Mechanistic Interpretability of GPT-2: Lexical and Contextual Layers in Sentiment Analysis

arXiv 2512.06681 ↗First posted 7 Dec 2025cs.CL cs.AIcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

We present a mechanistic interpretability study of GPT-2 that causally examines how sentiment information is processed across its transformer layers. Using systematic activation patching across all 12 layers, we test the hypothesized two-stage sentiment architecture comprising early lexical detection and mid-layer contextual integration. Our experiments confirm that early layers (0-3) act as lexical sentiment detectors, encoding stable, position specific polarity signals that are largely independent of context. However, all three contextual integration hypotheses: Middle Layer Concentration, Phenomenon Specificity, and Distributed Processing are falsified. Instead of mid-layer specialization, we find that contextual phenomena such as negation, sarcasm, domain shifts etc. are integrated primarily in late layers (8-11) through a unified, non-modular mechanism. These experimental findings provide causal evidence that GPT-2's sentiment computation differs from the predicted hierarchical pattern, highlighting the need for further empirical characterization of contextual integration in large language models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 9 of its 16 references that are in the map. First posted on arXiv 7 Dec 2025.

Builds on

9 of 16 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models2 Jul
  2. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching28 Nov

Other references · 7

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  3. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  4. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  5. 2019Designing and Interpreting Probes with Control Taskscited by 640
  6. 2024How to use and interpret activation patchingcited by 214
  7. 2018Adversarial Removal of Demographic Attributes from Text Datacited by 170

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Interpreting Negation in GPT-2: Layer- and Head-Level Causal Analysis
  2. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic review
  3. 2026Negative Before Positive: Asymmetric Valence Processing in Large Language Models
  4. 2025Mechanistic Interpretability of GPT-like Models on Summarization Tasks
  5. 2025From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?
  6. 2026Some Emotions Run Deeper: Layer-wise Probing and Causal Intervention in Large Language Models
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
  8. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis