mlsanityData to 7 Oct 2026

MUSE: A Unified Agentic Harness for MLLMs

arXiv 2606.03005 ↗First posted 2 Jun 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Despite rapid progress, multimodal large language models (MLLMs) still fail on tasks that humans solve effortlessly, such as navigating a grid maze from a screenshot or selecting the correct puzzle piece. Rather than retraining the model, we ask a complementary question: how much capability can be elicited from a frozen MLLM purely by improving the execution scaffold around it? We introduce MUSE, a multimodal unified structured execution harness that wraps any off-the-shelf MLLM with composable modules for task representation, visual processing, perception tool use, structured parsing, deterministic verification, and verifier-guided repair, without any model retraining. We evaluate MUSE across diverse benchmarks spanning visual spatial planning, visual perception, multimodal reasoning, and fine-grained visual discrimination, using multiple state-of-the-art MLLMs. MUSE delivers consistent gains over the bare model in all settings, with the largest jumps on challenging instances. Further analysis reveals that many MLLM failures arise from harness-level shortcomings rather than fundamental model deficits, and can be addressed through verifier-guided repair without touching the model. These findings highlight the agentic multimodal harness as a critical yet underexplored design dimension, offering an orthogonal avenue for improving MLLMs beyond model-centric optimization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 62 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

53 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Visual Reasoning through Tool-supervised Reinforcement Learning21 Apr
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models27 Mar
  3. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Models10 Dec
  4. 2024VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents12 Aug

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  9. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
37 more
  1. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  2. 2025OpenAI GPT-5 System Cardcited by 910
  3. 2023Large Language Models as Optimizerscited by 830
  4. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  5. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  8. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  9. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  10. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  11. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  13. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  14. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  15. 2026Meta-Harness: End-to-End Optimization of Model Harnessescited by 248
  16. 2025Machine Mental Imagery: Empower Multimodal Reasoning with Latent Visual Tokenscited by 120
  17. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  18. 2025Latent Visual Reasoningcited by 99
  19. 2025Monet: Reasoning in Latent Visual Space Beyond Images and Languagecited by 75
  20. 2026AutoHarness: improving LLM agents by automatically synthesizing a code harnesscited by 63
  21. 2025Feedback Descent: Open-Ended Text Optimization via Pairwise Comparisoncited by 42
  22. 2026Code as Agent Harnesscited by 40
  23. 2025TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoningcited by 34
  24. 2024CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Modelscited by 29
  25. 2026MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluationcited by 23
  26. 2026Adapting the Interface, Not the Model: Runtime Harness Adaptation for Deterministic LLM Agentscited by 22
  27. 2025VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Modelscited by 12
  28. 2026ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Modelcited by 12
  29. 2025Representation Potentials of Foundation Models for Multimodal Alignment: A Surveycited by 10
  30. 2026Ref-Adv: Exploring MLLM Visual Reasoning in Referring Expression Taskscited by 8
  31. 2025Aha Moment Revisited: Are VLMs Truly Capable of Self Verification in Inference-time Scaling?cited by 7
  32. 2025CoT Referring: Improving Referring Expression Tasks with Grounded Reasoningcited by 5
  33. 2025Restore-R1: Efficient Image Restoration Agents via Reinforcement Learning with Multimodal LLM Perceptual Feedbackcited by 4
  34. 2026Action Draft and Verify: A Self-Verifying Framework for Vision-Language-Action Modelcited by 4
  35. 2026The Indra Representation Hypothesis for Multimodal Alignmentcited by 4
  36. 2025Out-of-Sight Embodied Agents: Multimodal Tracking, Sensor Fusion, and Trajectory Forecastingcited by 1
  37. 2026HeavySkill: Heavy Thinking as the Inner Skill in Agentic Harnesscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agents
  2. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  3. 2025ToolScope: An Agentic Framework for Vision-Guided and Long-Horizon Tool Use
  4. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
  5. 2026VTC-Bench: Evaluating Agentic Multimodal Models via Compositional Visual Tool Chaining
  6. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents
  7. 2025From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
  8. 2024PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain