mlsanityData to 14 Oct 2026

ORIC: Benchmarking Object Recognition under Contextual Incongruity in Large Vision-Language Models

arXiv 2509.15695 ↗First posted 19 Sep 2025cs.CV cs.LGcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large Vision-Language Models (LVLMs) excel at captioning, visual question answering, and robotics by combining vision and language, yet they often miss obvious objects or hallucinate nonexistent ones in atypical scenes. We examine these failures through the lens of uncertainty, focusing on contextual incongruity, where objects appear unexpectedly or fail to appear in expected contexts, and show that such cases increase recognition difficulty for state-of-the-art LVLMs. To study this regime, we introduce the Object Recognition in Incongruous Context (ORIC) framework, which constructs incongruous object-context pairs through two complementary strategies: (1) LLM-guided sampling to identify hard-to-recognize objects present in the image and (2) CLIP-guided sampling to mine plausible but absent ones. Applied to MSCOCO, ORIC creates ORIC-Bench and ORIC-style training data. Evaluating 18 LVLMs and 2 open-vocabulary detectors reveals significant degradation and bias under incongruous contexts. Visual Reinforcement Fine-Tuning of Qwen3-VL-8B-Instruct on 600 ORIC samples improves performance on ORIC-Bench, AMBER, and HallusionBench. Overall, we show that contextual incongruity is a key source of uncertainty and provide tools for more reliable LVLMs. The dataset and code are publicly available at https://github.com/ZhaoyangLi-1/ORIC.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 68 of its 89 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

68 of 89 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Escaping the SpuriVerse: Can Large Vision-Language Models Generalize Beyond Seen Spurious Correlations?23 Jun
  2. 2025Kimi-VL Technical Report10 Apr
  3. 2024MACAROON: Training Vision-Language Models To Be Your Engaged Partners20 Jun
  4. 2024Unveiling Encoder-Free Vision-Language Models17 Jun
  5. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?29 Mar
  6. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding30 Nov
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  8. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
38 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  10. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  11. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  13. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  14. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  15. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  16. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  17. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  18. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  19. 2018Object Hallucination in Image Captioningcited by 669
  20. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  21. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 583
  22. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
  23. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  24. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  25. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  26. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  27. 2025Why Language Models Hallucinatecited by 220
  28. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 193
  29. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 190
  30. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  31. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  32. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129
  33. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 125
  34. 2024Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detectioncited by 122
  35. 2022CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailmentcited by 113
  36. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 102
  37. 2023Octopus: Embodied Vision-Language Programmer from Environmental Feedbackcited by 73
  38. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 72

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 21
  2. 2026Personalized Generative Models for Contextual Debiasingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: From Evaluation to Diagnosis
  2. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  3. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  4. 2025Object Detection with Multimodal Large Vision-Language Models: An In-depth Review
  5. 2024UOUO: Uncontextualized Uncommon Objects for Measuring Knowledge Horizons of Vision Language Models
  6. 2023Contextual Object Detection with Multimodal Large Language Models
  7. 2026C-GAP: Class-Aware and Online Prompting Improves Vision-Language Models on Imbalanced Classes
  8. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts