# Counterfactual Segmentation Reasoning: Diagnosing and Mitigating Pixel-Grounding Hallucination

arXiv: https://arxiv.org/abs/2506.21546 · HTML: https://mlsanity.com/paper/2506.21546/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination detection and mitigation, one of 17 directions in LLM reasoning.
- Builds on 10 of its 59 references that are in the map.
- First posted on arXiv 26 Jun 2025.

## Placed in

- LLM reasoning / [Hallucination detection and mitigation](https://mlsanity.com/lr/hallucination-detection-and-mitigation/)
- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 10 of 10)

- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement](https://mlsanity.com/paper/2503.06520/) (2025)
- [InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions](https://mlsanity.com/paper/2305.18047/) (2023)
- [VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning](https://mlsanity.com/paper/2505.12081/) (2025)
- [HyperSeg: Towards Universal Visual Segmentation with Large Language Model](https://mlsanity.com/paper/2411.17606/) (2024)
- [HallE-Control: Controlling Object Hallucination in Large Multimodal Models](https://mlsanity.com/paper/2310.01779/) (2023)
- [InstructSeg: Unifying Instructed Visual Segmentation with Multi-modal Large Language Models](https://mlsanity.com/paper/2412.14006/) (2024)
- [PRIMA: Multi-Image Vision-Language Models for Reasoning Segmentation](https://mlsanity.com/paper/2412.15209/) (2024)
- [Visual Jenga: Discovering Object Dependencies via Counterfactual Inpainting](https://mlsanity.com/paper/2503.21770/) (2025)

## Built on by (most cited first, 7 of 7)

- [VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidence](https://mlsanity.com/paper/2605.20676/) (2026)
- [3D-VCD: Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive Decoding](https://mlsanity.com/paper/2604.08645/) (2026)
- [DreamPartGen: Semantically Grounded Part-Level 3D Generation via Collaborative Latent Denoising](https://mlsanity.com/paper/2603.19216/) (2026)
- [From Pixels to Concepts: Do Segmentation Models Understand What They Segment?](https://mlsanity.com/paper/2605.09591/) (2026)
- [CFCamo: A Counterfactual Detect-or-Abstain Framework for Camouflaged Object Detection](https://mlsanity.com/paper/2606.11231/) (2026)
- [ChronoVision: Temporal Reasoning via Latent State Reconstruction](https://mlsanity.com/paper/2608.05631/) (2026)
- [Consistency Has a Computable Blind Spot: A Commutation Theory of Label-Free Reliability for Vision-Language Figure Reading](https://mlsanity.com/paper/2608.05675/) (2026)

## Abstract (arXiv)

Segmentation Vision-Language Models (VLMs) have significantly advanced grounded visual understanding, yet they remain prone to pixel-grounding hallucinations, producing masks for incorrect objects or for objects that are entirely absent. Existing evaluations rely almost entirely on text- or label-based perturbations, which check only whether the predicted mask matches the queried label. Such evaluations overlook the spatial footprint and severity of hallucination and therefore fail to reveal vision-driven hallucinations, which are more challenging and more prevalent. To address this gap, we formalize the task of Counterfactual Segmentation Reasoning (CSR), where a model must segment the referenced object in the factual image and abstain in its counterfactual counterpart. To support this task, we curate HalluSegBench, the first large-scale benchmark to diagnose referring and reasoning expression segmentation hallucinations using controlled visual counterfactuals, alongside new evaluation metrics that measure hallucination severity and disentangle vision- and language-driven failure modes. We further introduce RobustSeg, a segmentation VLM trained with counterfactual fine-tuning (CFT) to learn when to segment and when to abstain. Experimental results confirm RobustSeg reduces hallucinations by 30%, while improving segmentation performance on FP-RefCOCO(+/g).