mlsanityData to 14 Oct 2026

MoHoBench: Assessing Honesty of Multimodal Large Language Models via Unanswerable Visual Questions

arXiv 2507.21503 ↗First posted 29 Jul 2025cs.AI cs.CVcited by 11

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recently Multimodal Large Language Models (MLLMs) have achieved considerable advancements in vision-language tasks, yet produce potentially harmful or untrustworthy content. Despite substantial work investigating the trustworthiness of language models, MMLMs' capability to act honestly, especially when faced with visually unanswerable questions, remains largely underexplored. This work presents the first systematic assessment of honesty behaviors across various MLLMs. We ground honesty in models' response behaviors to unanswerable visual questions, define four representative types of such questions, and construct MoHoBench, a large-scale MMLM honest benchmark, consisting of 12k+ visual question samples, whose quality is guaranteed by multi-stage filtering and human verification. Using MoHoBench, we benchmarked the honesty of 28 popular MMLMs and conducted a comprehensive analysis. Our findings show that: (1) most models fail to appropriately refuse to answer when necessary, and (2) MMLMs' honesty is not solely a language modeling issue, but is deeply influenced by visual information, necessitating the development of dedicated methods for multimodal honesty alignment. Therefore, we implemented initial alignment methods using supervised and preference learning to improve honesty behavior, providing a foundation for future work on trustworthy MLLMs. Our data and code can be found at https://github.com/yanxuzhu/MoHoBench.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 69 references that are in the map. First posted on arXiv 29 Jul 2025.

Builds on

33 of 69 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs3 Mar
  2. 2025MM-RLHF: The Next Step Forward in Multimodal LLM Alignment14 Feb
  3. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  4. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization15 Nov
  5. 2024LLaVA-Critic: Learning to Evaluate Multimodal Models3 Oct
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  7. 2023Silkie: Preference Distillation for Large Visual Language Models17 Dec
  8. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep

Other references · 25

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  10. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  11. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  12. 2022A Survey on In-context Learningcited by 721
13 more
  1. 2018Object Hallucination in Image Captioningcited by 669
  2. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  3. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  4. 2021Truthful AI: Developing and governing AI that does not liecited by 113
  5. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  6. 2022Reliable Visual Question Answering: Abstain Rather Than Answer Incorrectlycited by 89
  7. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuningcited by 69
  8. 2024VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignmentcited by 58
  9. 2023Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledgecited by 32
  10. 2023ToViLaG: Your Visual-Language Generative Model is Also An Evildoercited by 27
  11. 2023UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Modelscited by 25
  12. 2024Visual Robustness Benchmark for Visual Question Answering (VQA)cited by 12
  13. 2024Embedding an Ethical Mind: Aligning Text-to-Image Synthesis via Lightweight Value Optimizationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 9.

11 papers.

  1. 2025Inflated Excellence or True Performance? Rethinking Medical Diagnostic Benchmarks with Dynamic Evaluationcited by 5
  2. 2025Counterfactual Reasoning for Steerable Pluralistic Value Alignment of Large Language Modelscited by 5
  3. 2026HypeMed: Enhancing Medication Recommendations with Hypergraph-Based Patient Relationshipscited by 3
  4. 2026Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMscited by 2
  5. 2026MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Modelscited by 0
  6. 2026Controllable Affective Generation via Latent Vector Steeringcited by 0
  7. 2026MMPCBench: Benchmarking Multimodal Large Language Models on Proactive Critique of Flawed Inputscited by 0
  8. 2026Knowing What Not to Answer: Selective Non-Compliance in Vision-Language Modelscited by 0
  9. 2026From Static Personal Values to Contextualized Personalization: Bayesian Personalized Value Alignment for LLMscited by 0
  10. 2026When Does an Image Determine the Answer? Benchmarking Visual Answerability across Charts and Scenescited by 0
  11. 2026Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIPcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
  2. 2026VirtueBench: Evaluating Trustworthiness under Uncertainty in Long Video Understanding
  3. 2024MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation
  4. 2025Measuring Epistemic Humility in Multimodal Large Language Models
  5. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios
  6. 2024A Survey on the Honesty of Large Language Models
  7. 2024MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal Large Language Models
  8. 2025VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering