# Holistic Evaluation for Interleaved Text-and-Image Generation

arXiv: https://arxiv.org/abs/2406.14643 · HTML: https://mlsanity.com/paper/2406.14643/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Generative Multimodal Models are In-Context Learners](https://mlsanity.com/paper/2312.13286/) (2023)
- [Towards a Unified Multi-Dimensional Evaluator for Text Generation](https://mlsanity.com/paper/2210.07197/) (2022)
- [Generating Images with Multimodal Language Models](https://mlsanity.com/paper/2305.17216/) (2023)
- [AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://mlsanity.com/paper/2402.12226/) (2024)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [Subject-driven Text-to-Image Generation via Apprenticeship Learning](https://mlsanity.com/paper/2304.00186/) (2023)
- [VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluation](https://mlsanity.com/paper/2312.14867/) (2023)
- [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024)
- [Visual Storytelling](https://mlsanity.com/paper/1604.03968/) (2016)
- [GPT-4V(ision) as a Generalist Evaluator for Vision-Language Tasks](https://mlsanity.com/paper/2311.01361/) (2023)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer](https://mlsanity.com/paper/2401.10208/) (2024)
- [ImagenHub: Standardizing the evaluation of conditional image generation models](https://mlsanity.com/paper/2310.01596/) (2023)
- [WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences](https://mlsanity.com/paper/2406.11069/) (2024)
- [Artificial Intelligence in the Creative Industries: A Review](https://mlsanity.com/paper/2007.12391/) (2020)
- [LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing](https://mlsanity.com/paper/2311.00571/) (2023)
- [Visual Goal-Step Inference using wikiHow](https://mlsanity.com/paper/2104.05845/) (2021)
- [X-Eval: Generalizable Multi-aspect Text Evaluation via Augmented Instruction Tuning with Auxiliary Evaluation Aspects](https://mlsanity.com/paper/2311.08788/) (2023)
- [A Pipeline for Creative Visual Storytelling](https://mlsanity.com/paper/1807.08077/) (2018)

## Built on by (most cited first, 28 of 28)

- [Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities](https://mlsanity.com/paper/2505.02567/) (2025)
- [LEGO-Puzzles: How Good Are MLLMs at Multi-Step Spatial Reasoning?](https://mlsanity.com/paper/2503.19990/) (2025)
- [Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback](https://mlsanity.com/paper/2412.15838/) (2024)
- [Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answering](https://mlsanity.com/paper/2505.16470/) (2025)
- [MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2410.10139/) (2024)
- [OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation](https://mlsanity.com/paper/2411.18499/) (2024)
- [LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models](https://mlsanity.com/paper/2504.10430/) (2025)
- [Multimodal RewardBench 2: Evaluating Omni Reward Models for Interleaved Text and Image](https://mlsanity.com/paper/2512.16899/) (2025)
- [How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformation](https://mlsanity.com/paper/2503.09598/) (2025)
- [UniM: A Unified Any-to-Any Interleaved Multimodal Benchmark](https://mlsanity.com/paper/2603.05075/) (2026)
- [Modality-Specialized Synergizers for Interleaved Vision-Language Generalists](https://mlsanity.com/paper/2407.03604/) (2024)
- [UEval: A Benchmark for Unified Multimodal Generation](https://mlsanity.com/paper/2601.22155/) (2026)
- [MetaScientist: A Human-AI Synergistic Framework for Automated Mechanical Metamaterial Design](https://mlsanity.com/paper/2412.16270/) (2024)
- [LLM-I: LLMs are Naturally Interleaved Multimodal Creators](https://mlsanity.com/paper/2509.13642/) (2025)
- [Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Training](https://mlsanity.com/paper/2603.25706/) (2026)
- [InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback](https://mlsanity.com/paper/2505.23950/) (2025)
- [RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answering](https://mlsanity.com/paper/2512.05119/) (2025)
- [Navigating Ideation Space: Decomposed Conceptual Representations for Positioning Scientific Ideas](https://mlsanity.com/paper/2601.08901/) (2026)
- [Towards Unified Multimodal Interleaved Generation via Group Relative Policy Optimization](https://mlsanity.com/paper/2603.09538/) (2026)
- [Deep-Reporter: Deep Research for Grounded Multimodal Long-Form Generation](https://mlsanity.com/paper/2604.10741/) (2026)
- [A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation](https://mlsanity.com/paper/2506.09427/) (2025)
- [DuoGen: Towards General Purpose Interleaved Multimodal Generation](https://mlsanity.com/paper/2602.00508/) (2026)
- [Pareto LoRA: Mitigating Modality Imbalance in Unified Multimodal Models via Pareto-Optimal Gradient Integration](https://mlsanity.com/paper/2606.17296/) (2026)
- [Modal Aphasia: Can Unified Multimodal Models Describe Images From Memory?](https://mlsanity.com/paper/2510.21842/) (2025)
- [ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generation](https://mlsanity.com/paper/2603.29902/) (2026)
- [COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts](https://mlsanity.com/paper/2604.27389/) (2026)
- [C3-Bench: A Context-Aware Change Captioning Benchmark](https://mlsanity.com/paper/2606.25445/) (2026)
- [What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation](https://mlsanity.com/paper/2609.37317/) (2026)

## Abstract (arXiv)

Interleaved text-and-image generation has been an intriguing research direction, where the models are required to generate both images and text pieces in an arbitrary order. Despite the emerging advancements in interleaved generation, the progress in its evaluation still significantly lags behind. Existing evaluation benchmarks do not support arbitrarily interleaved images and text for both inputs and outputs, and they only cover a limited number of domains and use cases. Also, current works predominantly use similarity-based metrics which fall short in assessing the quality in open-ended scenarios. To this end, we introduce InterleavedBench, the first benchmark carefully curated for the evaluation of interleaved text-and-image generation. InterleavedBench features a rich array of tasks to cover diverse real-world use cases. In addition, we present InterleavedEval, a strong reference-free metric powered by GPT-4o to deliver accurate and explainable evaluation. We carefully define five essential evaluation aspects for InterleavedEval, including text quality, perceptual quality, image coherence, text-image coherence, and helpfulness, to ensure a comprehensive and fine-grained assessment. Through extensive experiments and rigorous human evaluation, we show that our benchmark and metric can effectively evaluate the existing models with a strong correlation with human judgments surpassing previous reference-based metrics. We also provide substantial findings and insights to foster future research in interleaved generation and its evaluation.