mlsanityData to 7 Oct 2026

CLIP-UP: CLIP-Based Unanswerable Problem Detection for Visual Question Answering

arXiv 2501.01371 ↗First posted 2 Jan 2025cs.CVcited by 6

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) demonstrate remarkable capabilities in visual understanding and reasoning, such as in Visual Question Answering (VQA), where the model is asked a question related to a visual input. Still, these models can make distinctly unnatural errors, for example, providing (wrong) answers to unanswerable VQA questions, such as questions asking about objects that do not appear in the image. To address this issue, we propose CLIP-UP: CLIP-based Unanswerable Problem detection, a novel lightweight method for equipping VLMs with the ability to withhold answers to unanswerable questions. CLIP-UP leverages CLIP-based similarity measures to extract question-image alignment information to detect unanswerability, requiring efficient training of only a few additional layers, while keeping the original VLMs' weights unchanged. Tested across several models, CLIP-UP achieves significant improvements on benchmarks assessing unanswerability in both multiple-choice and open-ended VQA, surpassing other methods, while preserving original performance on other tasks.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 52 of its 58 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

52 of 58 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions5 Oct
  2. 2024MyVLM: Personalizing VLMs for User-Specific Queries21 Mar
  3. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations6 May

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2020Supervised Contrastive Learningcited by 2,803
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
37 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2015VQA: Visual Question Answeringcited by 2,489
  3. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  13. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  14. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  15. 2018Object Hallucination in Image Captioningcited by 661
  16. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  17. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  18. 2021Editing Factual Knowledge in Language Modelscited by 548
  19. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  20. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  21. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  22. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  23. 2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
  24. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  25. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  26. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  27. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  28. 2017An Analysis of Visual Question Answering Algorithmscited by 90
  29. 2023Concept Decomposition for Visual Exploration and Inspirationcited by 61
  30. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Promptscited by 47
  31. 2023Can We Edit Multimodal Large Language Models?cited by 44
  32. 2023Improving Automatic VQA Evaluation Using Large Language Modelscited by 37
  33. 2024Question Aware Vision Transformer for Multimodal Reasoningcited by 32
  34. 2024VLKEB: A Large Vision-Language Model Knowledge Editing Benchmarkcited by 30
  35. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 23
  36. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  37. 2024Unsolvable Problem Detection: Robust Understanding Evaluation for Large Multimodal Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 4.

6 papers.

  1. 2026AQUA-Bench: Beyond Finding Answers to Knowing When There Are None in Audio Question Answeringcited by 6
  2. 2025VisionTrap: Unanswerable Questions On Visual Datacited by 1
  3. 2025A Matter of Time: Revealing the Structure of Time in Vision-Language Modelscited by 0
  4. 2026Adjustable Text-Guided Backdoor Attacks with Natural-Word Triggers on Multimodal Pretrained Modelscited by 0
  5. 2026MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Modelscited by 0
  6. 2026From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey
  2. 2025Exploring Advanced Techniques for Visual Question Answering: A Comprehensive Comparison
  3. 2023Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering
  4. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities
  5. 2022CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks
  6. 2025Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection
  7. 2024A Comprehensive Survey on Visual Question Answering Datasets and Algorithms
  8. 2023Improving Zero-shot Visual Question Answering via Large Language Models with Reasoning Question Prompts