SegCompass: Exploring Interpretable Alignment with Sparse Autoencoders for Enhanced Reasoning Segmentation
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
While large language models provide strong compositional reasoning, existing reasoning segmentation pipelines fail to transparently connect this reasoning to visual perception. Current methods, such as latent query alignment, are end-to-end yet opaque "black boxes". Conversely, textual localization readout is merely readable, not truly interpretable, often functioning as an unconstrained post-hoc step. To bridge this interpretability gap, we propose SegCompass, an end-to-end model that leverages a Sparse Autoencoder (SAE) to forge an explicit, interpretable, and differentiable alignment pathway. Given an image-instruction pair, SegCompass first generates a chain-of-thought (CoT) trace. The core of our method is an SAE that maps both the CoT and visual tokens into a shared, high-dimensional sparse concept space. A query codebook selects salient concepts from this space, which are then spatially grounded by a slot mapper into a multi-slot heatmap that guides the final mask decoder. The entire model is trained jointly, unifying reinforcement learning for the reasoning path with standard segmentation supervision. This SAE-driven interface provides a "white-box" connection that is significantly more traceable than latent queries and more coherent than textual readouts. Extensive experiments on five challenging benchmarks demonstrate that SegCompass matches or surpasses state-of-the-art performance. Crucially, our visual and quantitative analyses show a strong correlation between the quality of the learned sparse concepts and final mask accuracy, confirming that SegCompass achieves superior results through its enhanced and inspectable alignment. Code is available at https://github.com/ZhenyuLU-Heliodore/SegCompass.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 63 references that are in the map. First posted on arXiv 21 May 2026.
Builds on
25 of 63 references.
Closest ancestors · 6
Same direction, newest first
- 2025UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning22 Sep
- 2025Seg-R1: Segmentation Can Be Surprisingly Simple with Reinforcement Learning27 Jun
- 2025SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning28 May
- 2025VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning17 May
- 2025Multimodal Chain-of-Thought Reasoning: A Comprehensive Survey16 Mar
- 2025Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement9 Mar
Other references · 19
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
- 2024Scaling and evaluating sparse autoencoderscited by 530
- 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 114
- 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 86
- 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
- 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
- 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 43
- 2025SAIF: A Sparse Autoencoder Framework for Interpreting and Steering Instruction Following of Language Modelscited by 32
7 more
- 2025X-SAM: From Segment Anything to Any Segmentationcited by 30
- 2025UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interfacecited by 27
- 2025Multimodal Referring Segmentation: A Surveycited by 22
- 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoderscited by 21
- 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
- 2025Towards Unified Referring Expression Segmentation Across Omni-Level Visual Target Granularitiescited by 5
- 2025Refer to Any Segmentation Mask Group With Vision-Language Promptscited by 5
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perceptioncited by 1
- 2026Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMscited by 0
- 2026Revisit to Segment: Working Memory Distillation for Reasoning Segmentationcited by 0
- 2026ContourVLA: A Closed-Loop Perception-Action Contour Policy for Generalized Referring Expression Segmentationcited by 0
- 2026ContourVLA: A Closed-Loop Perception-Action Contour Policy for Generalized Referring Expression Segmentation8 Oct 2026
- 2026Revisit to Segment: Working Memory Distillation for Reasoning Segmentation28 Sep 2026
- 2026Balancing Efficiency and Efficacy: Training-Free Attention-Guided Switching Between Explicit and Latent Thoughts for MLLMs4 Aug 2026
- 2026CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception22 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026Seg-Agent: Test-Time Multimodal Reasoning for Training-Free Language-Guided Segmentation
- 2026CR-Seg: Attention-Guided and CoT-Enhanced Coarse-to-Refined Reasoning Segmentation
- 2026Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models
- 2026Interpretability Transfer from Language to Vision via Sparse Autoencoders
- 2026GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine
- 2026AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation
- 2026Evaluating the Interpretability of Sparse Autoencoders with Concept Annotations
- 2026Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search