Universal Boosts, Specific Suppressors: Sparse Autoencoder Steering of Medical Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Medical vision-language models
Abstract (arXiv)
Medical vision-language models (VLMs) often hallucinate findings when generating chest X-ray reports: they fabricate findings that are not present in the image, miss important ones, or locate them incorrectly. We mitigate this without weight updates by decoding-time residual steering on a per-token sparse autoencoder (SAE) basis: Top-K SAEs on late layers, causal steering against clinical errors, then combined suppress/boost intervention at inference time. On the MIMIC-CXR test split, our inference-only method improves the quality of generated reports for three radiology VLMs (RadVLM, LLaVA-Rad, and CheXOne), with relative improvements of +5.4%, +7.2%, and +17.0% in the clinical composite metric, and statistically significant GREEN gains on all backbones. A cross-model feature alignment shows that the quality-promoting (boost) directions overlap strongly across architectures, whereas hallucination-linked (suppress) directions are model-specific. Therefore, transferable steering must treat suppression per-backbone, rather than sharing a universal suppress list. The same recipe transfers zero-shot to IU-Xray (Green +7.7% rel.) without retraining, confirming that the identified features are properties of the model, not of the training corpus. We release causal feature sets and an interactive feature dashboard: https://cxr-sparse-feature-dashboard.netlify.app/.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 35 references that are in the map. First posted on arXiv 24 May 2026.
Builds on
15 of 35 references.
Closest ancestors · 1
Same direction, newest first
Other references · 14
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
- 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
- 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
- 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
- 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
- 2024GREEN: Generative Radiology Report Evaluation and Error Notationcited by 124
- 2024Steering Language Model Refusal with Sparse Autoencoderscited by 89
- 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
- 2024Steering Large Language Models using Conceptors: Improving Addition-Based Activation Engineeringcited by 33
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering
- 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation
- 2026RadJEPA: Radiology Encoder for Chest X-Rays via Joint Embedding Predictive Architecture
- 2025Revolutionizing Radiology Workflow with Factual and Efficient CXR Report Generation
- 2026GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations
- 2026U-VLM: Hierarchical Vision Language Modeling for Report Generation
- 2024CXR-Agent: Vision-language models for chest X-ray interpretation with uncertainty aware radiology reporting
- 2026CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imaging