# MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs

arXiv: https://arxiv.org/abs/2410.12332 · HTML: https://mlsanity.com/paper/2410.12332/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 88 of its 133 references that are in the map.
- First posted on arXiv 16 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 88)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)

## Built on by (most cited first, 30 of 39)

- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation](https://mlsanity.com/paper/2502.11903/) (2025)
- [Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models](https://mlsanity.com/paper/2501.05767/) (2025)
- [ContextGen: Contextual Layout Anchoring for Identity-Consistent Multi-Instance Generation](https://mlsanity.com/paper/2510.11000/) (2025)
- [MM-Gesture: Towards Precise Micro-Gesture Recognition through Multimodal Fusion](https://mlsanity.com/paper/2507.08344/) (2025)
- [Towards Cross-View Point Correspondence in Vision-Language Models](https://mlsanity.com/paper/2512.04686/) (2025)
- [Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning](https://mlsanity.com/paper/2507.00748/) (2025)
- [MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems](https://mlsanity.com/paper/2503.16549/) (2025)
- [LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models](https://mlsanity.com/paper/2505.15616/) (2025)
- [More Images, More Problems? A Controlled Analysis of VLM Failure Modes](https://mlsanity.com/paper/2601.07812/) (2026)
- [MedSG-Bench: A Benchmark for Medical Image Sequences Grounding](https://mlsanity.com/paper/2505.11852/) (2025)
- [Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering](https://mlsanity.com/paper/2601.05159/) (2026)
- [From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignment](https://mlsanity.com/paper/2503.20472/) (2025)
- [MMLU-Reason: Benchmarking Multi-Task Multi-modal Language Understanding and Reasoning](https://mlsanity.com/paper/2505.16459/) (2025)
- [Watch Before You Answer: Learning from Visually Grounded Post-Training](https://mlsanity.com/paper/2604.05117/) (2026)
- [BVINet: Unlocking Blind Video Inpainting with Zero Annotations](https://mlsanity.com/paper/2502.01181/) (2025)
- [TangramPuzzle: Evaluating Multimodal Large Language Models with Compositional Spatial Reasoning](https://mlsanity.com/paper/2601.16520/) (2026)
- [SlideAgent: Hierarchical Agentic Framework for Multi-Page Visual Document Understanding](https://mlsanity.com/paper/2510.26615/) (2025)
- [MOAT: Evaluating LMMs for Capability Integration and Instruction Grounding](https://mlsanity.com/paper/2503.09348/) (2025)
- [Clear Nights Ahead: Towards Multi-Weather Nighttime Image Restoration](https://mlsanity.com/paper/2505.16479/) (2025)
- [MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoning](https://mlsanity.com/paper/2603.12266/) (2026)
- [CVT-Bench: Probing Spatial-State Integrity through Counterfactual Viewpoint Transformations](https://mlsanity.com/paper/2603.21114/) (2026)
- [Pseudo-Labeling Driven Refinement of Benchmark Object Detection Datasets via Analysis of Learning Patterns](https://mlsanity.com/paper/2506.00997/) (2025)
- [KnowDR-REC: A Benchmark for Referring Expression Comprehension with Real-World Knowledge](https://mlsanity.com/paper/2508.14080/) (2025)
- [GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language Models](https://mlsanity.com/paper/2601.04777/) (2026)
- [V-Reflection: Transforming MLLMs from Passive Observers to Active Interrogators](https://mlsanity.com/paper/2604.03307/) (2026)
- [Seeing Before Agreeing: Aligning Multi-Agent Consensus with Visual Evidence](https://mlsanity.com/paper/2605.30698/) (2026)
- [One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs](https://mlsanity.com/paper/2606.14277/) (2026)
- [Mixture-of-Thought-Tokens: Unifying Perception and Reasoning for Free-form Multimodal Grounding](https://mlsanity.com/paper/2607.24407/) (2026)
- [ColLab: A Collaborative Spatial Progressive Data Engine for Referring Expression Comprehension and Generation](https://mlsanity.com/paper/2509.23955/) (2025)

## Abstract (arXiv)

While multimodal large language models (MLLMs) have demonstrated extraordinary vision-language understanding capabilities, their abilities to solve instance-level visual-language problems beyond a single image warrant further exploration. To assess these unproven abilities of MLLMs, this paper proposes a new visual grounding task called multi-context visual grounding, which aims to localize instances of interest across multiple images based on open-ended text prompts. In order to facilitate this research, we construct a new dataset MC-Bench that features 2K high-quality and manually annotated samples. Each sample consists of an instance-level labeled image pair and a corresponding text prompt that indicates the target instances in the images. These text prompts are highly open-ended and follow three distinct styles, covering 20 practical skills. We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities, along with our developed simple yet effective agentic baseline and a finetuned baseline by multi-context instruction tuning. Our evaluation reveals a non-trivial performance gap between existing MLLMs and humans, along with some insightful observations that suggest potential future directions. We hope that MC-Bench and our empirical findings encourage the research community to further advance the untapped potentials of MLLMs in instance-level tasks, particularly in multi-image contexts. Project page: https://xuyunqiu.github.io/MC-Bench.