mlsanityData to 7 Oct 2026

See, Say, and Segment: Teaching LMMs to Overcome False Premises

arXiv 2312.08366 ↗First posted 13 Dec 2023cs.CVcited by 40

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Current open-source Large Multimodal Models (LMMs) excel at tasks such as open-vocabulary language grounding and segmentation but can suffer under false premises when queries imply the existence of something that is not actually present in the image. We observe that existing methods that fine-tune an LMM to segment images significantly degrade their ability to reliably determine ("see") if an object is present and to interact naturally with humans ("say"), a form of catastrophic forgetting. In this work, we propose a cascading and joint training approach for LMMs to solve this task, avoiding catastrophic forgetting of previous skills. Our resulting model can "see" by detecting whether objects are present in an image, "say" by telling the user if they are not, proposing alternative queries or correcting semantic errors in the query, and finally "segment" by outputting the mask of the desired objects if they exist. Additionally, we introduce a novel False Premise Correction benchmark dataset, an extension of existing RefCOCO(+/g) referring segmentation datasets (which we call FP-RefCOCO(+/g)). The results show that our method not only detects false premises up to 55% better than existing approaches, but under false premise conditions produces relative cIOU improvements of more than 31% over baselines, and produces natural language feedback judged helpful up to 67% of the time.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 73 of its 97 references that are in the map. First posted on arXiv 13 Dec 2023.

Builds on

73 of 97 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  2. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  4. 2023Contextual Object Detection with Multimodal Large Language Models29 May

Other references · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Segment Anythingcited by 8,158
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2017Mask R-CNNcited by 7,259
  10. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,845
  11. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  12. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
38 more
  1. 2016Pyramid Scene Parsing Networkcited by 2,472
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  4. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,421
  7. 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,148
  8. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,128
  9. 2016Modeling Context in Referring Expressionscited by 1,054
  10. 2018Dual Attention Network for Scene Segmentationcited by 1,001
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  13. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  14. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  15. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  16. 2018Panoptic Segmentationcited by 796
  17. 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 717
  18. 2018Object Hallucination in Image Captioningcited by 661
  19. 2015Learning Deconvolution Network for Semantic Segmentationcited by 365
  20. 2019Panoptic-DeepLabcited by 362
  21. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  22. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  23. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  24. 2022Generalized Decoding for Pixel, Image, and Languagecited by 292
  25. 2017ICNet for Real-Time Semantic Segmentation on High-Resolution Imagescited by 261
  26. 2016Segmentation from Natural Language Expressionscited by 257
  27. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  28. 2021Vision-Language Transformer and Query Generation for Referring Segmentationcited by 239
  29. 2021K-Net: Towards Unified Image Segmentationcited by 211
  30. 2020Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentationcited by 204
  31. 2019Asymmetric Non-local Neural Networks for Semantic Segmentationcited by 196
  32. 2015ParseNet: Looking Wider to See Bettercited by 176
  33. 2019UPSNet: A Unified Panoptic Segmentation Networkcited by 175
  34. 2023PACO: Parts and Attributes of Common Objectscited by 146
  35. 2023BuboGPT: Enabling Visual Grounding in Multi-Modal LLMscited by 114
  36. 2021Semi-supervised Semantic Segmentation with Directional Context-aware Consistencycited by 112
  37. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 110
  38. 2014From Captions to Visual Concepts and Backcited by 109

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 13; 2026: 13.

40 papers.

Sort:

By area:

  1. 2023GSVA: Generalized Segmentation via Multimodal Large Language Modelscited by 153
  2. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  3. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  4. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  5. 2024LLARVA: Vision-Action Instruction Tuning Enhances Robot Learningcited by 87
  6. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Modelcited by 70
  7. 2024LaSagnA: Language-based Segmentation Assistant for Complex Queriescited by 55
  8. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 37
  9. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 32
  10. 2024Reason3D: Searching and Reasoning 3D Segmentation via Large Language Modelcited by 27
  11. 2024SegLLM: Multi-round Reasoning Segmentationcited by 25
  12. 2025Multimodal Referring Segmentation: A Surveycited by 22
28 more
  1. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  2. 2025LISAT: Language-Instructed Segmentation Assistant for Satellite Imagerycited by 13
  3. 2025Reasoning Segmentation for Images and Videos: A Surveycited by 12
  4. 2024DeiSAM: Segment Anything with Deictic Promptingcited by 9
  5. 2025Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Predictioncited by 7
  6. 2024SPIN: Hierarchical Segmentation with Subpart Granularity in Natural Imagescited by 5
  7. 2025CoT Referring: Improving Referring Expression Tasks with Grounded Reasoningcited by 5
  8. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMscited by 5
  9. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentationcited by 4
  10. 2025RESAnything: Attribute Prompting for Arbitrary Referring Segmentationcited by 4
  11. 2026ROSE: Retrieval-Oriented Segmentation Enhancementcited by 3
  12. 2026VISTAQA: Benchmarking Joint Visual Question Answering and Pixel-Level Evidencecited by 3
  13. 2024EditScout: Locating Forged Regions from Diffusion-based Edited Images with Multimodal LLMcited by 2
  14. 2024Reasoning to Attend: Try to Understand How <SEG> Token Workscited by 2
  15. 2025Judge Before Answer: Can MLLM Discern the False Premise in Question?cited by 2
  16. 2026ProactiveBench: Benchmarking Proactiveness in Multimodal Large Language Modelscited by 2
  17. 2026AnchorSeg: Language Grounded Query Banks for Reasoning Segmentationcited by 1
  18. 2026What to Edit Next: Visually Aligned Image-Editing Follow-Up Suggestions in Conversational Systemscited by 1
  19. 2025LTCA: Long-range Temporal Context Attention for Referring Video Object Segmentationcited by 0
  20. 2025Seg the HAB: Language-Guided Geospatial Algae Bloom Reasoning and Segmentationcited by 0
  21. 2026S3POT: Contrast-Driven Face Occlusion Segmentation via Self-Supervised Prompt Learningcited by 0
  22. 2026MedVeriSeg: Teaching LISA-Like Medical Segmentation Models to Verify Query Validity Without Extra Trainingcited by 0
  23. 2026CFCamo: A Counterfactual Detect-or-Abstain Framework for Camouflaged Object Detectioncited by 0
  24. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  25. 2026Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentationcited by 0
  26. 2026Position Is All You Need: A Free Lunch Token Compression Strategy for MLLM-based Referring Expression Segmentationcited by 0
  27. 2026Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Modelscited by 0
  28. 2026Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Deflection and Hallucination in Large Vision-Language Models
  2. 2026MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models
  3. 2025See, Think, Learn: A Self-Taught Multimodal Reasoner
  4. 2024F-LMM: Grounding Frozen Large Multimodal Models
  5. 2025LaRe: Latent Refocusing for Multimodal Reasoning
  6. 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
  7. 2026ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models
  8. 2026Retrieved Images as Visual Thought: Training-Free Multimodal In-Context Learning for the Open-vs-Closed Gap