# MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs

arXiv: https://arxiv.org/abs/2407.01509 · HTML: https://mlsanity.com/paper/2407.01509/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 57 of its 66 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)

## Builds on (most cited first, 30 of 57)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension](https://mlsanity.com/paper/2307.16125/) (2023)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [Yi: Open Foundation Models by 01.AI](https://mlsanity.com/paper/2403.04652/) (2024)

## Built on by (most cited first, 30 of 62)

- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models](https://mlsanity.com/paper/2407.11691/) (2024)
- [Aria: An Open Multimodal Native Mixture-of-Experts Model](https://mlsanity.com/paper/2410.05993/) (2024)
- [MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers](https://mlsanity.com/paper/2508.14704/) (2025)
- [InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model](https://mlsanity.com/paper/2501.12368/) (2025)
- [MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2411.15296/) (2024)
- [MMEvol: Empowering Multimodal Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2409.05840/) (2024)
- [MM-IFEngine: Towards Multimodal Instruction Following](https://mlsanity.com/paper/2504.07957/) (2025)
- [Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback](https://mlsanity.com/paper/2412.15838/) (2024)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference](https://mlsanity.com/paper/2502.18411/) (2025)
- [A Survey on Multimodal Benchmarks: In the Era of Large AI Models](https://mlsanity.com/paper/2409.18142/) (2024)
- [Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.15910/) (2025)
- [Ming-Flash-Omni: A Sparse, Unified Architecture for Multimodal Perception and Generation](https://mlsanity.com/paper/2510.24821/) (2025)
- [JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation](https://mlsanity.com/paper/2410.17250/) (2024)
- [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://mlsanity.com/paper/2506.01413/) (2025)
- [MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks](https://mlsanity.com/paper/2507.19634/) (2025)
- [Generative RLHF-V: Learning Principles from Multi-modal Human Preference](https://mlsanity.com/paper/2505.18531/) (2025)
- [Constructive Distortion: Improving MLLMs with Attention-Guided Image Warping](https://mlsanity.com/paper/2510.09741/) (2025)
- [Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models](https://mlsanity.com/paper/2511.02650/) (2025)
- [ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models](https://mlsanity.com/paper/2410.04659/) (2024)
- [Omni-Reward: Towards Generalist Omni-Modal Reward Modeling with Free-Form Preferences](https://mlsanity.com/paper/2510.23451/) (2025)
- [Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization](https://mlsanity.com/paper/2506.04039/) (2025)
- [MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm](https://mlsanity.com/paper/2512.02895/) (2025)
- [INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning](https://mlsanity.com/paper/2412.03565/) (2024)
- [When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorations](https://mlsanity.com/paper/2505.24449/) (2025)
- [Steering Multimodal Large Language Models Decoding for Context-Aware Safety](https://mlsanity.com/paper/2509.19212/) (2025)
- [Empowering Reliable Visual-Centric Instruction Following in MLLMs](https://mlsanity.com/paper/2601.03198/) (2026)
- [SO-Bench: A Structural Output Evaluation of Multimodal LLMs](https://mlsanity.com/paper/2511.21750/) (2025)
- [Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitration](https://mlsanity.com/paper/2602.03677/) (2026)

## Abstract (arXiv)

We introduce MIA-Bench, a new benchmark designed to evaluate multimodal large language models (MLLMs) on their ability to strictly adhere to complex instructions. Our benchmark comprises a diverse set of 400 image-prompt pairs, each crafted to challenge the models' compliance with layered instructions in generating accurate responses that satisfy specific requested patterns. Evaluation results from a wide array of state-of-the-art MLLMs reveal significant variations in performance, highlighting areas for improvement in instruction fidelity. Additionally, we create extra training data and explore supervised fine-tuning to enhance the models' ability to strictly follow instructions without compromising performance on other tasks. We hope this benchmark not only serves as a tool for measuring MLLM adherence to instructions, but also guides future developments in MLLM training methods.