mlsanityData to 7 Oct 2026

SO-Bench: A Structural Output Evaluation of Multimodal LLMs

arXiv 2511.21750 ↗First posted 23 Nov 2025cs.CV cs.AI cs.CLcited by 5

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

Multimodal large language models (MLLMs) are increasingly deployed in real-world, agentic settings where outputs must not only be correct, but also conform to predefined data schemas. Despite recent progress in structured generation in textual domain, there is still no benchmark that systematically evaluates schema-grounded information extraction and reasoning over visual inputs. In this work, we conduct a comprehensive study of visual structural output capabilities for MLLMs with our carefully designed SO-Bench benchmark. Covering four visual domains, including UI screens, natural images, documents, and charts, SO-Bench is built from over 6.5K diverse JSON schemas and 1.8K curated image-schema pairs with human-verified quality. Benchmarking experiments on open-sourced and frontier proprietary models reveal persistent gaps in predicting accurate, schema compliant outputs, highlighting the need for better multimodal structured reasoning. Beyond benchmarking, we further conduct training experiments to largely improve the model's structured output capability. We make the benchmark and evaluation publicly available at https://github.com/apple/ml-sobench

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 58 references that are in the map. First posted on arXiv 23 Nov 2025.

Builds on

50 of 58 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs3 Mar
  2. 2024Pixtral 12B9 Oct
  3. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning30 Sep
  4. 2024LLaVA-OneVision: Easy Visual Task Transfer6 Aug
  5. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs1 Jul
  6. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs24 Jun

Other references · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2025Gemma 3 Technical Reportcited by 1,686
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  8. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  9. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  10. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  11. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  12. 2016Modeling Context in Referring Expressionscited by 1,054
32 more
  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
  2. 2022Exploring Plain Vision Transformer Backbones for Object Detectioncited by 723
  3. 2021InfographicVQAcited by 430
  4. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  5. 2025Kimi-VL Technical Reportcited by 381
  6. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  7. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  8. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  9. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  10. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  11. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 165
  12. 2024OmniDocBench: Benchmarking Diverse PDF Document Parsing with Comprehensive Annotationscited by 144
  13. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  14. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 111
  15. 2025MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsingcited by 108
  16. 2025ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answeringcited by 105
  17. 2020Mirror Descent Policy Optimizationcited by 102
  18. 2022Towards End-to-End Unified Scene Text Detection and Layout Analysiscited by 96
  19. 2025PaddleOCR-VL: Boosting Multilingual Document Parsing via a 0.9B Ultra-Compact Vision-Language Modelcited by 84
  20. 2024CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacycited by 80
  21. 2024GTA: A Benchmark for General Tool Agentscited by 77
  22. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 66
  23. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  24. 2023WebUI: A Dataset for Enhancing Visual UI Understanding with Web Semanticscited by 45
  25. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Modelscited by 30
  26. 2025StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputscited by 23
  27. 2024Image2Struct: Benchmarking Structure Extraction for Vision-Language Modelscited by 18
  28. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17
  29. 2023Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?cited by 16
  30. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  31. 2024StrucText-Eval: Evaluating Large Language Model's Reasoning Ability in Structure-Rich Textcited by 13
  32. 2025ToolVQA: A Dataset for Multi-step Reasoning VQA with External Toolscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026VAREX: A Benchmark for Multi-Modal Structured Extraction from Documentscited by 4
  2. 2026MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agentscited by 2
  3. 2026DACA-GRPO: Denoising-Aware Credit Assignment for Reinforcement Learning in Diffusion Language Modelscited by 1
  4. 2026S2SServiceBench: A Multimodal Benchmark for Last-Mile S2S Climate Servicescited by 0
  5. 2026HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  2. 2024Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
  3. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
  4. 2026TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs
  5. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  6. 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?
  7. 2024A Survey on Benchmarks of Multimodal Large Language Models
  8. 2025LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models