# The Importance of Prompt Tuning for Automated Neuron Explanations

arXiv: https://arxiv.org/abs/2310.06200 · HTML: https://mlsanity.com/paper/2310.06200/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 9 Oct 2023.

## Builds on (most cited first, 14 of 14)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Towards Expert-Level Medical Question Answering with Large Language Models](https://mlsanity.com/paper/2305.09617/) (2023)
- [FinGPT: Open-Source Financial Large Language Models](https://mlsanity.com/paper/2306.06031/) (2023)
- [CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks](https://mlsanity.com/paper/2204.10965/) (2022)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Neuron-level Interpretation of Deep NLP Models: A Survey](https://mlsanity.com/paper/2108.13138/) (2021)
- [Legal Prompting: Teaching a Language Model to Think Like a Lawyer](https://mlsanity.com/paper/2212.01326/) (2022)
- [On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?](https://mlsanity.com/paper/2303.11146/) (2023)
- [Towards Applying Powerful Large AI Models in Classroom Teaching: Opportunities, Challenges and Prospects](https://mlsanity.com/paper/2305.03433/) (2023)

## Built on by (most cited first, 12 of 12)

- [Automatically Interpreting Millions of Features in Large Language Models](https://mlsanity.com/paper/2410.13928/) (2024)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [Linear Explanations for Individual Neurons](https://mlsanity.com/paper/2405.06855/) (2024)
- [Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework](https://mlsanity.com/paper/2506.15538/) (2025)
- [Effective Skill Unlearning through Intervention and Abstention](https://mlsanity.com/paper/2503.21730/) (2025)
- [Crafting Large Language Models for Enhanced Interpretability](https://mlsanity.com/paper/2407.04307/) (2024)
- [AND: Audio Network Dissection for Interpreting Deep Acoustic Models](https://mlsanity.com/paper/2406.16990/) (2024)
- [Constructing Interpretable Features from Compositional Neuron Groups](https://mlsanity.com/paper/2506.10920/) (2025)
- [Evaluating Neuron Explanations: A Unified Framework with Sanity Checks](https://mlsanity.com/paper/2506.05774/) (2025)
- [Interpreting Language Models Through Concept Descriptions: A Survey](https://mlsanity.com/paper/2510.01048/) (2025)
- [Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligence](https://mlsanity.com/paper/2608.12036/) (2026)
- [Key Path Identification for Resolving Knowledge Conflicts via SAE-based Steering](https://mlsanity.com/paper/2609.08173/) (2026)

## Abstract (arXiv)

Recent advances have greatly increased the capabilities of large language models (LLMs), but our understanding of the models and their safety has not progressed as fast. In this paper we aim to understand LLMs deeper by studying their individual neurons. We build upon previous work showing large language models such as GPT-4 can be useful in explaining what each neuron in a language model does. Specifically, we analyze the effect of the prompt used to generate explanations and show that reformatting the explanation prompt in a more natural way can significantly improve neuron explanation quality and greatly reduce computational cost. We demonstrate the effects of our new prompts in three different ways, incorporating both automated and human evaluations.