# Sparse Visual Thought Circuits in Vision-Language Models

arXiv: https://arxiv.org/abs/2603.25075 · HTML: https://mlsanity.com/paper/2603.25075/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 15 of its 18 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)
- Mechanistic interpretability / [Interpreting vision and multimodal models](https://mlsanity.com/mechinterp/interpreting-vision-and-multimodal-models/)

## Builds on (most cited first, 15 of 15)

- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning](https://mlsanity.com/paper/2403.16999/) (2024)
- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [Towards Interpreting Visual Information Processing in Vision-Language Models](https://mlsanity.com/paper/2410.07149/) (2024)
- [SAEs Are Good for Steering -- If You Select the Right Features](https://mlsanity.com/paper/2505.20063/) (2025)
- [Improving Vision Transformers by Revisiting High-frequency Components](https://mlsanity.com/paper/2204.00993/) (2022)
- [What's in the Image? A Deep-Dive into the Vision of Vision Language Models](https://mlsanity.com/paper/2411.17491/) (2024)
- [Interpreting CLIP with Hierarchical Sparse Autoencoders](https://mlsanity.com/paper/2502.20578/) (2025)
- [Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation](https://mlsanity.com/paper/2511.05923/) (2025)
- [SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Models](https://mlsanity.com/paper/2510.26769/) (2025)
- [Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Models](https://mlsanity.com/paper/2511.09809/) (2025)
- [Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding](https://mlsanity.com/paper/2512.19070/) (2025)
- [Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Models](https://mlsanity.com/paper/2511.10292/) (2025)

## Abstract (arXiv)

Sparse autoencoders (SAEs) improve interpretability in multimodal models, but it remains unclear whether SAE features form modular, composable units for reasoning-an assumption underlying many intervention-based steering methods. We test this modularity hypothesis and find it often fails: intervening on a task-selective feature set can modestly improve reasoning accuracy, while intervening on the union of two such sets reliably induces output drift (large unintended changes in predictions) and degrades accuracy, even under norm-matched perturbations. This non modular circuit interference is consistent with shared internal pathways where feature unions amplify activation shifts. We develop a reproducible causal pipeline to localize and test these sparse visual thought circuits in Qwen3-VL-8B. On a controlled synthetic benchmark with seven task types and three difficulty levels, linear probes identify a mid decoder locus for task type information. We train SAEs at this layer, construct task-selective sets via an explicit rule, and perform inference time scaling and ablation while quantifying accuracy and drift. Our findings-validated with bootstrapped subsamples and permutation controls, and replicated across multiple VLM families and five diverse datasets clarify the boundaries of SAE feature composability and provide a rigorous diagnostic framework for more reliable VLM control.