# Multimodal Neurons in Pretrained Text-Only Transformers

arXiv: https://arxiv.org/abs/2308.01544 · HTML: https://mlsanity.com/paper/2308.01544/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Pretrained Transformers as Universal Computation Engines](https://mlsanity.com/paper/2103.05247/) (2021)
- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [Interpreting Deep Visual Representations via Network Dissection](https://mlsanity.com/paper/1711.05611/) (2017)
- [Language Models are General-Purpose Interfaces](https://mlsanity.com/paper/2206.06336/) (2022)
- [MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning](https://mlsanity.com/paper/2112.05253/) (2021)
- [Plansformer: Generating Symbolic Plans using Transformers](https://mlsanity.com/paper/2212.08681/) (2022)
- [LM-Debugger: An Interactive Tool for Inspection and Intervention in Transformer-Based Language Models](https://mlsanity.com/paper/2204.12130/) (2022)

## Built on by (most cited first, 30 of 49)

- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Visual In-Context Learning for Large Vision-Language Models](https://mlsanity.com/paper/2402.11574/) (2024)
- [Towards Interpreting Visual Information Processing in Vision-Language Models](https://mlsanity.com/paper/2410.07149/) (2024)
- [Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations](https://mlsanity.com/paper/2410.02762/) (2024)
- [Cross-modal Information Flow in Multimodal Large Language Models](https://mlsanity.com/paper/2411.18620/) (2024)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Vision Transformers Don't Need Trained Registers](https://mlsanity.com/paper/2506.08010/) (2025)
- [Frozen Transformers in Language Models Are Effective Visual Encoder Layers](https://mlsanity.com/paper/2310.12973/) (2023)
- [A Concept-Based Explainability Framework for Large Multimodal Models](https://mlsanity.com/paper/2406.08074/) (2024)
- [Finding and Editing Multi-Modal Neurons in Pre-Trained Transformers](https://mlsanity.com/paper/2311.07470/) (2023)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [MMNeuron: Discovering Neuron-Level Domain-Specific Interpretation in Multimodal Large Language Model](https://mlsanity.com/paper/2406.11193/) (2024)
- [How Visual Representations Map to Language Feature Space in Multimodal LLMs](https://mlsanity.com/paper/2506.11976/) (2025)
- [Concept-skill Transferability-based Data Selection for Large Vision-Language Models](https://mlsanity.com/paper/2406.10995/) (2024)
- [Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs](https://mlsanity.com/paper/2405.16700/) (2024)
- [TACO: Enhancing Multimodal In-context Learning via Task Mapping-Guided Sequence Configuration](https://mlsanity.com/paper/2505.17098/) (2025)
- [Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training](https://mlsanity.com/paper/2509.26625/) (2025)
- [Cross-Modal Projection in Multimodal LLMs Doesn't Really Project Visual Attributes to Textual Space](https://mlsanity.com/paper/2402.16832/) (2024)
- [Neuron-Level Sequential Editing for Large Language Models](https://mlsanity.com/paper/2410.04045/) (2024)
- [Analyzing Finetuning Representation Shift for Multimodal LLMs Steering](https://mlsanity.com/paper/2501.03012/) (2025)
- [Attribution Analysis Meets Model Editing: Advancing Knowledge Correction in Vision Language Models with VisEdit](https://mlsanity.com/paper/2408.09916/) (2024)
- [DNA: Uncovering Universal Latent Forgery Knowledge](https://mlsanity.com/paper/2601.22515/) (2026)
- [Automatic Discovery of Visual Circuits](https://mlsanity.com/paper/2404.14349/) (2024)
- [VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set](https://mlsanity.com/paper/2510.21323/) (2025)
- [Object Recognition as Next Token Prediction](https://mlsanity.com/paper/2312.02142/) (2023)
- [When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models](https://mlsanity.com/paper/2507.13868/) (2025)
- [Where Culture Fades: Revealing the Cultural Gap in Text-to-Image Generation](https://mlsanity.com/paper/2511.17282/) (2025)
- [Arbitration Failure, Not Perceptual Blindness: How Vision-Language Models Resolve Visual-Linguistic Conflicts](https://mlsanity.com/paper/2604.09364/) (2026)

## Abstract (arXiv)

Language models demonstrate remarkable capacity to generalize representations learned in one modality to downstream tasks in other modalities. Can we trace this ability to individual neurons? We study the case where a frozen text transformer is augmented with vision using a self-supervised visual encoder and a single linear projection learned on an image-to-text task. Outputs of the projection layer are not immediately decodable into language describing image content; instead, we find that translation between modalities occurs deeper within the transformer. We introduce a procedure for identifying "multimodal neurons" that convert visual representations into corresponding text, and decoding the concepts they inject into the model's residual stream. In a series of experiments, we show that multimodal neurons operate on specific visual concepts across inputs, and have a systematic causal effect on image captioning.