mlsanityData to 14 Oct 2026

OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation

arXiv 2411.18499 ↗First posted 27 Nov 2024cs.CVcited by 31

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have made significant strides in visual understanding and generation tasks. However, generating interleaved image-text content remains a challenge, which requires integrated multimodal understanding and generation abilities. While the progress in unified models offers new solutions, existing benchmarks are insufficient for evaluating these methods due to limitations in data size and diversity. To bridge this gap, we introduce OpenING, a comprehensive benchmark comprising 5,400 high-quality human-annotated instances across 56 real-world tasks. OpenING covers diverse daily scenarios such as travel guide, design, and brainstorming, offering a robust platform for challenging interleaved generation methods. In addition, we present IntJudge, a judge model for evaluating open-ended multimodal generation methods. Trained with a novel data pipeline, our IntJudge achieves an agreement rate of 82.42% with human judgments, outperforming GPT-based evaluators by 11.34%. Extensive experiments on OpenING reveal that current interleaved generation methods still have substantial room for improvement. Key findings on interleaved image-text generation are further presented to guide the development of next-generation models.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 65 of its 94 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

65 of 94 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models14 Oct
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  3. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generation8 Jul
  4. 2024EVALALIGN: Supervised Fine-Tuning Multimodal LLMs with Human-Aligned Data for Evaluating Text-to-Image Models24 Jun
  5. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  6. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark7 Feb
  7. 2023Generative Multimodal Models are In-Context Learners20 Dec
  8. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation30 Nov
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)29 Sep
  10. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  12. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents21 Jun
1 more
  1. 2023Generating Images with Multimodal Language Models26 May

Other references · 52

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  12. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
38 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  2. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  3. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  4. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  5. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 795
  6. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  7. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  8. 2023Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecastingcited by 707
  9. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  10. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  11. 2019PubLayNet: largest dataset ever for document layout analysiscited by 351
  12. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
  13. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
  14. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  15. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  16. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
  17. 2020Detection and Tracking Meet Drones Challengecited by 263
  18. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  19. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  20. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 185
  21. 2016Visual Storytellingcited by 176
  22. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  23. 2024Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretrainingcited by 150
  24. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  25. 2024SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editingcited by 97
  26. 2024Towards Open-ended Visual Quality Comparisoncited by 85
  27. 2023Intelligent Grimm -- Open-ended Visual Storytelling via Latent Diffusion Modelscited by 77
  28. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 73
  29. 2024BoQ: A Place is Worth a Bag of Learnable Queriescited by 64
  30. 2022Large-scale Text-to-Image Generation Models for Visual Artists' Creative Workscited by 61
  31. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
  32. 2021Visual Goal-Step Inference using wikiHowcited by 43
  33. 2024OpenDataLab: Empowering General Artificial Intelligence with Open Datasetscited by 29
  34. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28
  35. 2023OpenLEAF: Open-Domain Interleaved Image-Text Generation and Evaluationcited by 16
  36. 2024K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferencescited by 16
  37. 2021A Hybrid Bandit Model with Visual Priors for Creative Ranking in Display Advertisingcited by 15
  38. 2024ARES: Alternating Reinforcement Learning and Supervised Fine-Tuning for Enhanced Multi-Modal Chain-of-Thought Reasoning Through Diverse AI Feedbackcited by 13

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 16.

31 papers.

Sort:

By area:

  1. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 166
  2. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 132
  3. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  4. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 36
  5. 2026SenseNova-U1: Unifying Multimodal Understanding and Generation with NEO-unify Architecturecited by 32
  6. 2025Uni-MMMU: A Massive Multi-discipline Multimodal Unified Benchmarkcited by 29
  7. 2025CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generationcited by 23
  8. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  9. 2025MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Modelscited by 15
  10. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Modelcited by 13
  11. 2026GENIUS: Generative Fluid Intelligence Evaluation Suitecited by 12
  12. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
19 more
  1. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 11
  2. 2026UEval: A Benchmark for Unified Multimodal Generationcited by 9
  3. 2026SenseNova-U1.5: Towards Native Unified Visual Intelligencecited by 7
  4. 2025LLM-I: LLMs are Naturally Interleaved Multimodal Creatorscited by 6
  5. 2025CATCH: A Modular Cross-domain Adaptive Template with Hookcited by 6
  6. 2026VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoningcited by 6
  7. 2026Wan-Weaver: Interleaved Multi-modal Generation via Decoupled Trainingcited by 5
  8. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedbackcited by 4
  9. 2025RAG-IGBench: Innovative Evaluation for RAG-based Interleaved Generation in Open-domain Question Answeringcited by 4
  10. 2026AEGIS: Exploring the Limit of World Knowledge Capabilities for Unified Mulitmodal Modelscited by 1
  11. 2026Unified Hallucination Fuzzing for Multimodal Large Language Modelscited by 1
  12. 2025Learning to See Through Flarecited by 0
  13. 2025Bringing The Consistency Gap: Explicit Structured Memory for Interleaved Image-Text Generationcited by 0
  14. 2025Constantly Improving Image Models Need Constantly Improving Benchmarkscited by 0
  15. 2026ATP-Bench: Towards Agentic Tool Planning for MLLM Interleaved Generationcited by 0
  16. 2026IMUG-Bench: Benchmarking Unified Multimodal Models on Interleaved Understanding and Generationcited by 0
  17. 2026C3-Bench: A Context-Aware Change Captioning Benchmarkcited by 0
  18. 2026Improving Generalization Robustness of Multimodal RLVRcited by 0
  19. 2026What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation
  2. 2023MiniGPT-5: Interleaved Vision-and-Language Generation via Generative Vokens
  3. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
  4. 2024The Revolution of Multimodal Large Language Models: A Survey
  5. 2026DuoGen: Towards General Purpose Interleaved Multimodal Generation
  6. 2026Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment
  7. 2025MMMG: a Comprehensive and Reliable Benchmark for Multitask Multimodal Generation
  8. 2025OmniMamba: Efficient and Unified Multimodal Understanding and Generation via State Space Models