mlsanityData to 7 Oct 2026

MEBench: A Novel Benchmark for Understanding Mutual Exclusivity Bias in Vision-Language Models

arXiv 2505.20122 ↗First posted 26 May 2025cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

This paper introduces MEBench, a novel benchmark for evaluating mutual exclusivity (ME) bias, a cognitive phenomenon observed in children during word learning. Unlike traditional ME tasks, MEBench further incorporates spatial reasoning to create more challenging and realistic evaluation settings. To facilitate controlled experimentation, we also present a flexible and scalable data generation pipeline that supports the construction of diverse annotated scenes. We assess the performance of various vision-language models (VLMs) on this benchmark using novel evaluation metrics that capture key aspects of ME-based reasoning. We find that these VLMs exhibit weak ME bias, while showing some ability to leverage extra spatial context to resolve ambiguity in multiple novel object settings. Project page: http://mebench.github.io/.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 39 references that are in the map. First posted on arXiv 26 May 2025.

Builds on

25 of 39 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos7 Jan
  2. 2024F-LMM: Grounding Frozen Large Multimodal Models9 Jun
  3. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  4. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  5. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  6. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  7. 2024YOLOv10: Real-Time End-to-End Object Detectioncited by 508
  8. 2022Kubric: A scalable dataset generatorcited by 373
  9. 2021Towards Open World Object Detectioncited by 297
  10. 2016Thingi10K: A Dataset of 10,000 3D-Printing Modelscited by 182
  11. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  12. 2024Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generationcited by 129
9 more
  1. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understandingcited by 126
  2. 2021Using Shape to Categorize: Low-Shot Learning with an Explicit Shape Biascited by 115
  3. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 64
  4. 2020Grounded Language Learning Fast and Slowcited by 54
  5. 2022Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactionscited by 34
  6. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
  7. 2022Learning Dense Object Descriptors from Multiple Views for Low-shot Category Generalizationcited by 6
  8. 2023LSFSL: Leveraging Shape Information in Few-shot Learningcited by 4
  9. 2023Low-shot Object Learning with Mutual Exclusivity Biascited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Repetition Without Exclusivity: Scale Sensitivity of Referential Mechanisms in Child-Scale Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DevBench: A multimodal developmental benchmark for language learning
  2. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  3. 2026Almieyar-Oryx-BloomBench: A Bilingual Multimodal Benchmark for Cognitively Informed Evaluation of Vision-Language Models
  4. 2026Scale Can't Overcome Pragmatics: The Impact of Reporting Bias on Vision-Language Reasoning
  5. 2025A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models
  6. 2024VLBiasBench: A Comprehensive Benchmark for Evaluating Bias in Large Vision-Language Model
  7. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  8. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models