mlsanityData to 14 Oct 2026

MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs

arXiv 2407.01509 ↗First posted 1 Jul 2024cs.CV cs.CLcited by 62

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

We introduce MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to strictly adhere to complex instructions. Our benchmark comprises a diverse set of 400 image-prompt pairs, each crafted to challenge the models' compliance with layered instructions in generating accurate responses that satisfy specific requested patterns. Evaluation results from a wide array of state-of-the-art MLLMs reveal significant variations in performance, highlighting areas for improvement in instruction fidelity. Additionally, we create extra training data and explore supervised fine-tuning to enhance the models' ability to strictly follow instructions without compromising performance on other tasks. We hope this benchmark not only serves as a tool for measuring MLLM adherence to instructions, but also guides future developments in MLLM training methods.

Key facts. Cited by 62 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 66 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

57 of 66 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models3 May
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites25 Apr
  3. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD9 Apr
  4. 2024How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts20 Feb
  5. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models8 Feb
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks21 Dec
  7. 2023Generative Multimodal Models are In-Context Learners20 Dec
  8. 2023VILA: On Pre-training for Visual Language Models12 Dec
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  10. 2023InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models20 Nov
  11. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models13 Nov
  12. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration7 Nov
11 more
  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Aligning Large Multimodal Models with Factually Augmented RLHF25 Sep
  3. 2023Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages23 Aug
  4. 2023VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use12 Aug
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  8. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic27 Jun
  9. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  10. 2023Generating Images with Multimodal Language Models26 May
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  9. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  10. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  11. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  12. 2019Towards VQA Models That Can Readcited by 1,718
22 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  5. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  6. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  7. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
  8. 2024Yi: Open Foundation Models by 01.AIcited by 719
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  10. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  11. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 501
  12. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuningcited by 445
  13. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  14. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  15. 2024What matters when building vision-language models?cited by 326
  16. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 256
  17. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  18. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  19. 2023mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understandingcited by 168
  20. 2023LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Modelscited by 113
  21. 2024Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Modelscited by 97
  22. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 27; 2026: 24.

62 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  3. 2024Aria: An Open Multimodal Native Mixture-of-Experts Modelcited by 138
  4. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
  5. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 74
  6. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  7. 2024MMEvol: Empowering Multimodal Large Language Models with Evol-Instructcited by 42
  8. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  9. 2024Align Anything: Training All-Modality Models to Follow Instructions with Language Feedbackcited by 39
  10. 2026STEP3-VL-10B Technical Reportcited by 32
  11. 2025OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preferencecited by 28
  12. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
38 more
  1. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Modelscited by 26
  2. 2025Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generationcited by 22
  3. 2024JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluationcited by 19
  4. 2025Incentivizing Reasoning for Advanced Instruction-Following of Large Language Modelscited by 17
  5. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  6. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  7. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  8. 2025Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Modelscited by 13
  9. 2024ActiView: Evaluating Active Perception Ability for Multimodal Large Language Modelscited by 11
  10. 2025Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferencescited by 11
  11. 2025Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimizationcited by 10
  12. 2025MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigmcited by 9
  13. 2024INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuningcited by 8
  14. 2025When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorationscited by 7
  15. 2025Steering Multimodal Large Language Models Decoding for Context-Aware Safetycited by 7
  16. 2026Empowering Reliable Visual-Centric Instruction Following in MLLMscited by 7
  17. 2025SO-Bench: A Structural Output Evaluation of Multimodal LLMscited by 6
  18. 2026Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitrationcited by 6
  19. 2025BlueLM-2.5-3B Technical Reportcited by 5
  20. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  21. 2026MMKU-Bench: A Multimodal Update Benchmark for Diverse Visual Knowledgecited by 5
  22. 2025IFEval-Audio: Benchmarking Instruction-Following Capability in Audio-based Large Language Modelscited by 4
  23. 2025Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editingcited by 4
  24. 2026EVE: Verifiable Self-Evolution of MLLMs via Executable Visual Transformationscited by 4
  25. 2025SeeingEye: Agentic Information Flow Unlocks Multimodal Reasoning In Text-only LLMscited by 3
  26. 2025HAIBU-ReMUD: Reasoning Multimodal Ultrasound Dataset and Model Bridging to General Specific Domainscited by 2
  27. 2025Multimodal Policy Internalization for Conversational Agentscited by 2
  28. 2026MM-CondChain: A Programmatically Verified Benchmark for Visually Grounded Deep Compositional Reasoningcited by 2
  29. 2026Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Mergingcited by 2
  30. 2026VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Followingcited by 2
  31. 2026ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agentscited by 1
  32. 2026MMCL-Bench: Multimodal Context Learning from Visual Rules, Procedures, and Evidencecited by 1
  33. 2026DMN: A Compositional Framework for Jailbreaking Multimodal LLMs with Multi-Image Inputscited by 1
  34. 2026Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluationcited by 1
  35. 2024Magnifier Prompt: Tackling Multimodal Hallucination via Extremely Simple Instructionscited by 0
  36. 2024From Simple to Professional: A Combinatorial Controllable Image Captioning Agentcited by 0
  37. 2025KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controlscited by 0
  38. 2025INTERLACE: Interleaved Layer Pruning and Efficient Adaptation in Large Vision-Language Modelscited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models
  2. 2024CoMMIT: Coordinated Multimodal Instruction Tuning
  3. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  4. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  5. 2024Towards Robust Instruction Tuning on Multimodal Large Language Models
  6. 2023Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning
  7. 2024Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data
  8. 2023On the Performance of Multimodal Language Models