mlsanityData to 14 Oct 2026

ISO-Bench: Benchmarking Multimodal Causal Reasoning in Visual-Language Models through Procedural Plans

arXiv 2507.23135 ↗First posted 30 Jul 2025cs.CLcited by 2

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Understanding causal relationships across modalities is a core challenge for multimodal models operating in real-world environments. We introduce ISO-Bench, a benchmark for evaluating whether models can infer causal dependencies between visual observations and procedural text. Each example presents an image of a task step and a text snippet from a plan, with the goal of deciding whether the visual step occurs before or after the referenced text step. Evaluation results on ten frontier vision-language models show underwhelming performance: the best zero-shot F1 is only 0.57, and chain-of-thought reasoning yields only modest gains (up to 0.62 F1), largely behind humans (0.98 F1). Our analysis further highlights concrete directions for improving causal understanding in multimodal models.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 7 of its 34 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

7 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models30 Oct

Other references · 6

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  2. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  3. 2024NATURAL PLAN: Benchmarking LLMs on Natural Language Planningcited by 128
  4. 2018Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Imagescited by 118
  5. 2020Hybrid Models for Open Set Recognitioncited by 109
  6. 2024CaT-BENCH: Benchmarking Language Model Understanding of Causal and Temporal Dependencies in Planscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?cited by 6
  2. 2026Evaluating Reasoning Fidelity in Visual Text Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Multimodal Causal Reasoning Benchmark: Challenging Vision Large Language Models to Discern Causal Links Across Modalities
  2. 2026See2Think: Do Multimodal Models Really Use Intermediate Visual States?
  3. 2026The Abstraction Gap in Vision-Language Causal Reasoning
  4. 2025CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models
  5. 2024JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
  6. 2026WOVEN: Weaving Visual World Modeling into Multimodal LLMs
  7. 2025ROVER: Benchmarking Reciprocal Cross-Modal Reasoning for Omnimodal Generation
  8. 2025MM-CoT:A Benchmark for Probing Visual Chain-of-Thought Reasoning in Multimodal Models