mlsanityData to 7 Oct 2026

Sparse Visual Thought Circuits in Vision-Language Models

arXiv 2603.25075 ↗First posted 26 Mar 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Sparse autoencoders (SAEs) improve interpretability in multimodal models, but it remains unclear whether SAE features form modular, composable units for reasoning-an assumption underlying many intervention-based steering methods. We test this modularity hypothesis and find it often fails: intervening on a task-selective feature set can modestly improve reasoning accuracy, while intervening on the union of two such sets reliably induces output drift (large unintended changes in predictions) and degrades accuracy, even under norm-matched perturbations. This non modular circuit interference is consistent with shared internal pathways where feature unions amplify activation shifts. We develop a reproducible causal pipeline to localize and test these sparse visual thought circuits in Qwen3-VL-8B. On a controlled synthetic benchmark with seven task types and three difficulty levels, linear probes identify a mid decoder locus for task type information. We train SAEs at this layer, construct task-selective sets via an explicit rule, and perform inference time scaling and ablation while quantifying accuracy and drift. Our findings-validated with bootstrapped subsamples and permutation controls, and replicated across multiple VLM families and five diverse datasets clarify the boundaries of SAE feature composability and provide a rigorous diagnostic framework for more reliable VLM control.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 18 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

15 of 18 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models26 Nov
  2. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  3. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning25 Mar

Other references · 12

  1. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  3. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  4. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  5. 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 75
  6. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  7. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 54
  8. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 22
  9. 2025SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Modelscited by 8
  10. 2025Test-Time Spectrum-Aware Latent Steering for Zero-Shot Generalization in Vision-Language Modelscited by 3
  11. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decodingcited by 2
  12. 2025Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Transcoders Trace Visual Grounding and Hallucinations in Vision-Language Models
  2. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities
  3. 2026SPARC: Separating Perception And Reasoning Circuits for Test-time Scaling of VLMs
  4. 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models
  5. 2026Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders
  6. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders
  7. 2025What's Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning
  8. 2025See, Think, Learn: A Self-Taught Multimodal Reasoner