mlsanityData to 7 Oct 2026

ROSE: Benchmarking the Perception-to-Action Gap in Multimodal Models

arXiv 2606.19965 ↗First posted 18 Jun 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) are increasingly expected to act on visual information, yet the same scene may require different actions under different task contexts. How reliably can a model turn the same visual evidence into the action required by the current context? To answer this question, we introduce textscROSE (Reference-conditioned Oddity and Symbolic Execution), a controlled benchmark that holds the visual scene fixed while varying region constraints and required symbolic outputs. Through coupled counting and coordinate-action tasks, textscROSE tests whether models can infer an implicit majority reference and act on the resulting fine-grained visual evidence under changing contexts. Across nine recent MLLMs, performance drops by as much as 44.5 percentage points from counting-oriented tasks to region-conditioned action, despite 98.8% human performance. The gap persists on paired scenes and regions for which the same model returns the correct count, while global-click and matched local controls show that coordinate grounding explains only part of the loss, revealing a distinct, model-dependent bottleneck in turning shared visual evidence into context-specific actions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 32 references that are in the map. First posted on arXiv 18 Jun 2026.

Builds on

27 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026OddGridBench: Exposing the Lack of Fine-Grained Visual Discrepancy Sensitivity in Multimodal Large Language Models10 Mar
  2. 2025Qwen3-VL Technical Report26 Nov
  3. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe16 Sep
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  5. 2024CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models21 Feb
  6. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug

Other references · 21

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024GPT-4o System Cardcited by 4,467
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  4. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  5. 2024VisionZip: Longer is Better but Not Necessary in Vision Language Modelscited by 298
  6. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  7. 2026Qwen3.5-Omni Technical Reportcited by 146
  8. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  9. 2026RynnBrain: Open Embodied Foundation Modelscited by 33
  10. 2025VGRP-Bench: Visual Grid Reasoning Puzzle Benchmark for Large Vision-Language Modelscited by 26
  11. 2025MME-Unify: A Comprehensive Benchmark for Unified Multimodal Understanding and Generation Modelscited by 24
  12. 2024ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Modelscited by 19
9 more
  1. 2025Vision-Language Models Can't See the Obviouscited by 13
  2. 2025RynnEC: Bringing MLLMs into Embodied Worldcited by 11
  3. 2026Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Taskscited by 8
  4. 2025S2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural Guidancecited by 5
  5. 2026CogFlow: Bridging Perception and Reasoning through Knowledge Internalization for Visual Mathematical Problem Solvingcited by 5
  6. 2026ChartNet: A Million-Scale, High-Quality Multimodal Dataset for Robust Chart Understandingcited by 5
  7. 2026Visual Reasoning Benchmark: Evaluating Multimodal LLMs on Classroom-Authentic Visual Problems from Primary Educationcited by 2
  8. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  9. 2026M3-ACE: Rectifying Visual Perception in Multimodal Math Reasoning via Multi-Agentic Context Engineeringcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models
  2. 2026See2Think: Do Multimodal Models Really Use Intermediate Visual States?
  3. 2023Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models
  4. 2026Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenes
  5. 2025ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual-Language Models through Procedural Plans
  6. 2026MM-OPD: Towards One More Bottleneck Between Perception and Reasoning
  7. 2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models
  8. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models