# Suppressing Prior-Comparison Hallucinations in Radiology Report Generation via Semantically Decoupled Latent Steering

arXiv: https://arxiv.org/abs/2602.23676 · HTML: https://mlsanity.com/paper/2602.23676/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Medical vision-language models](https://mlsanity.com/mllm/medical-vision-language-models/)
- Mechanistic interpretability / [Interpreting biomedical and science models](https://mlsanity.com/mechinterp/interpreting-biomedical-and-science-models/)

## Builds on (most cited first, 30 of 33)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison](https://mlsanity.com/paper/1901.07031/) (2019)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Vision-Language Models for Vision Tasks: A Survey](https://mlsanity.com/paper/2304.00685/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models](https://mlsanity.com/paper/2310.14566/) (2023)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [RadGraph: Extracting Clinical Entities and Relations from Radiology Reports](https://mlsanity.com/paper/2106.14463/) (2021)
- [Function Vectors in Large Language Models](https://mlsanity.com/paper/2310.15213/) (2023)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [On the Automatic Generation of Medical Imaging Reports](https://mlsanity.com/paper/1711.08195/) (2017)
- [Visual Interpretability for Deep Learning: a Survey](https://mlsanity.com/paper/1802.00614/) (2018)
- [TieNet: Text-Image Embedding Network for Common Thorax Disease Classification and Reporting in Chest X-rays](https://mlsanity.com/paper/1801.04334/) (2018)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation](https://mlsanity.com/paper/2303.10323/) (2023)
- [CheXpert Plus: Augmenting a Large Chest X-ray Dataset with Text Radiology Reports, Patient Demographics and Additional Image Formats](https://mlsanity.com/paper/2405.19538/) (2024)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors](https://mlsanity.com/paper/2210.06340/) (2022)
- [Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation](https://mlsanity.com/paper/2006.03744/) (2020)
- [Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation](https://mlsanity.com/paper/2206.01988/) (2022)
- [Discovering Bias in Latent Space: An Unsupervised Debiasing Approach](https://mlsanity.com/paper/2406.03631/) (2024)
- [Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Models](https://mlsanity.com/paper/2408.13808/) (2024)
- [Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining](https://mlsanity.com/paper/2304.14204/) (2023)
- [Towards Interpretable Counterfactual Generation via Multimodal Autoregression](https://mlsanity.com/paper/2503.23149/) (2025)
- [HC-LLM: Historical-Constrained Large Language Models for Radiology Report Generation](https://mlsanity.com/paper/2412.11070/) (2024)

## Built on by (most cited first, 1 of 1)

- [Translating Inference-Time Control to Radiology Vision-Language Models: Activation Steering for Pneumonia Classification on Chest X-rays](https://mlsanity.com/paper/2606.20852/) (2026)

## Abstract (arXiv)

Automated radiology report generation using vision-language models (VLMs) is limited by the risk of prior-comparison hallucination, where the model generates historical findings unsupported by the current study. We address this challenge with a training-free, inference-time control framework termed Semantically Decoupled Latent Steering (SDLS). Unlike generic activation steering, which often suffers from semantic entanglement, our approach constructs a semantic-free intervention vector via large language model (LLM)-driven semantic decomposition followed by QR-based orthogonalization. This orthogonalization step is critical. It leverages geometric constraints to filter out the clinical semantics often entangled in standard principal component analysis (PCA) directions, ensuring that the steering vector targets only the ``historical comparison" axis. We validate our method on the BiomedGPT foundation model, demonstrating that it overcomes the trade-off between hallucination suppression and clinical accuracy. Extensive experiments on MIMIC-CXR, and zero-shot transfer evaluation on CheXpert Plus and IU-Xray, demonstrate the robustness of our approach. Quantitative evaluations on MIMIC-CXR show that our approach significantly reduces the probability of historical hallucinations (FilBERT score decreases from 0.2373 to 0.1889) and improves clinical label fidelity (CheXpert macro-F1 increases from 0.2242 to 0.3208). Supplementary evaluations confirm that the structural integrity of the clinical narrative is maintained.