Measuring Concept Content in Text from LLM Activations: ESG Evidence from Concept Vectors and Linear Probes
On the map: Mechanistic interpretability → SAE features and their applications
Abstract (arXiv)
Recent work shows that Large Language Models (LLMs) encode more information in their activations than they show in their responses. We ask whether the information contained in activations could be used as a source of measurement for the presence of concepts. We compare two algorithms for extraction of concepts, i.e., Recursive Feature Machine (RFM) algorithm and linear probing against an embedding baseline, surface baselines, and the model's own response using a human-annotated Environmental, Social and Governance (ESG) dataset of financial texts. The best linear probe ranges from 0.6 to 3.3 percentage points below fine-tuned domain classifiers. The probes also outscore the model's own response in all comparisons. Furthermore, the probes outperform the RFM concept vectors in all accuracy comparisons. In a synthetic mixture test, their mean-pooled score tracks the number of concept sentences in a paragraph better than the concept-vector score. Put together, a linear probe on a frozen LLM is a cost-effective and generalizable alternative to fine-tuning a domain classifier.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 16 of its 26 references that are in the map. First posted on arXiv 7 Aug 2026.
Builds on
16 of 26 references.
References · 16
- 2017Attention Is All You Needcited by 60,361
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
- 2023The Internal State of an LLM Knows When It's Lyingcited by 659
- 2019Designing and Interpreting Probes with Control Taskscited by 640
- 2023Language Models Represent Space and Timecited by 316
- 2018The Geometry of Culture: Analyzing Meaning through Word Embeddingscited by 87
4 more
- 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
- 2025Toward universal steering and monitoring of AI modelscited by 30
- 2025xRFM: Accurate, scalable, and interpretable feature learning models for tabular datacited by 16
- 2024LLM-Measure: Generating Valid, Consistent, and Reproducible Text-Based Measures for Social Science Researchcited by 5
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Improving Preference Extraction In LLMs By Identifying Latent Knowledge Through Classifying Probes
- 2023From Text to Source: Results in Detecting Large Language Model-Generated Content
- 2026Text Analytics Evaluation Framework: A Case Study on LLMs and Social Media
- 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labels
- 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Data
- 2024Frame Representation Hypothesis: Multi-Token LLM Interpretability and Concept-Guided Text Generation
- 2024ResearchArena: Benchmarking Large Language Models' Ability to Collect and Organize Information as Research Agents
- 2025Beyond Tokens in Language Models: Interpreting Activations through Text Genre Chunks