mlsanityData to 14 Oct 2026

Seeing Is Not Deciding: Can Multimodal LLMs Act as Effective CEOs?

arXiv 2608.05864 ↗First posted 6 Aug 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Large language models are increasingly applied as autonomous decision-making agents. However, in executive business decisions, existing benchmarks are limited to textonly settings. This makes it unclear whether models can perceive visual business evidence and effectively integrate it to improve decision quality. We introduce C-SUITEBENCH, a controlled multimodal benchmark that includes five decision tasks under paired text-only and multimodal conditions across 50 scenarios. We place nine frontier models in the role of a chief executive officer and evaluate their decision-making ability. Multimodal inputs consistently improve evidence-centric reasoning, with the largest and most reliable gains appearing in risk forecasting and board-facing justification. However, we uncover a multimodal integration paradox: adding visual business information degrades constrained resource allocation for all nine models, even as visual grounding itself improves. Ablation experiments reveal that this failure emerges from signal crowding, although each visual channel helps individually, their combination disrupts constraint satisfaction during decoding. These findings demonstrate that visual perception and constrained action are separable bottlenecks in multimodal agents, and that indiscriminate visual augmentation can harm high-stakes decision making, motivating selective grounding strategies for future executive AI systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 35 references that are in the map. First posted on arXiv 6 Aug 2026.

Builds on

25 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 24

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  6. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  8. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  9. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  10. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  11. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
  12. 2019PlotQA: Reasoning over Scientific Plotscited by 321
12 more
  1. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  2. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 270
  3. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 196
  4. 2024ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoningcited by 68
  5. 2025Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Taskscited by 65
  6. 2025FinAgentBench: A Benchmark Dataset for Agentic Retrieval in Financial Question Answeringcited by 29
  7. 2025DSGBench: A Diverse Strategic Game Benchmark for Evaluating LLM-based Agents in Complex Decision-Making Environmentscited by 23
  8. 2026RealFin: How Well Do LLMs Reason About Finance When Users Leave Things Unsaid?cited by 9
  9. 2026RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environmentscited by 6
  10. 2026CEO-Bench: Can Agents Play the Long Game?cited by 6
  11. 2026FIRE: A Comprehensive Benchmark for Financial Intelligence and Reasoning Evaluationcited by 5
  12. 2026Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MBA: Multimodal Benchmark and Agents for Real-World Business Ideation
  2. 2026A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications
  3. 2025Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models
  4. 2025Multi-Crit: Benchmarking Multimodal Judges on Pluralistic Criteria-Following
  5. 2024MageBench: Bridging Large Multimodal Models to Agents
  6. 2026The Evidence Anchoring Boundary: Where Visual Interventions Matter and When They Help in Multimodal LLMs
  7. 2026See2Think: Do Multimodal Models Really Use Intermediate Visual States?
  8. 2026MM-OPD: Towards One More Bottleneck Between Perception and Reasoning