mlsanityData to 14 Oct 2026

Superscopes: Amplifying Internal Feature Representations for Language Model Interpretation

arXiv 2503.02078 ↗First posted 3 Mar 2025cs.CL cs.AI cs.LGcited by 3

On the map: Mechanistic interpretability → Knowledge and representations inside LLMs

Abstract (arXiv)

Understanding and interpreting the internal representations of large language models (LLMs) remains an open challenge. Patchscopes introduced a method for probing internal activations by patching them into new prompts, prompting models to self-explain their hidden representations. We introduce Superscopes, a technique that systematically amplifies superposed features in MLP outputs (multilayer perceptron) and hidden states before patching them into new contexts. Inspired by the "features as directions" perspective and the Classifier-Free Guidance (CFG) approach from diffusion models, Superscopes amplifies weak but meaningful features, enabling the interpretation of internal representations that previous methods failed to explain-all without requiring additional training. This approach provides new insights into how LLMs build context and represent complex concepts, further advancing mechanistic interpretability.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability. Builds on 23 of its 43 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

23 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models11 Jan
  2. 2023Eliciting Latent Predictions from Transformers with the Tuned Lens14 Mar

Other references · 21

  1. 2022Classifier-Free Diffusion Guidancecited by 5,847
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2022Toy Models of Superpositioncited by 971
  6. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  9. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 422
  10. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  11. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  12. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
9 more
  1. 2014Object Detectors Emerge in Deep Scene CNNscited by 179
  2. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  3. 2015Visualizing and Understanding Recurrent Networkscited by 160
  4. 2021Post-hoc Interpretability for Neural NLP: A Surveycited by 156
  5. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  6. 2023Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generationcited by 69
  7. 2023Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Modelscited by 60
  8. 2021BERTnesia: Investigating the capture and forgetting of knowledge in BERTcited by 40
  9. 2023The Hidden Language of Diffusion Modelscited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025The LLM Already Knows: Estimating LLM-Perceived Question Difficulty via Hidden Representationscited by 29
  2. 2026Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Modelscited by 3
  3. 2026Prompting Underestimates LLM Capability for Time Series Classificationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025InverseScope: Scalable Activation Inversion for Interpreting Large Language Models
  2. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Models
  3. 2026Mechanistic Knobs in LLMs: Retrieving and Steering High-Order Semantic Features via Sparse Autoencoders
  4. 2026ICA Lens: Interpreting Language Models Without Training Another Dictionary
  5. 2025Interpreting Language Models Through Concept Descriptions: A Survey
  6. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  7. 2026A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability
  8. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspective