mlsanityData to 7 Oct 2026

LMM4LMM: Benchmarking and Evaluating Large-multimodal Image Generation with LMMs

arXiv 2504.08358 ↗First posted 11 Apr 2025cs.CVcited by 24

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Recent breakthroughs in large multimodal models (LMMs) have significantly advanced both text-to-image (T2I) generation and image-to-text (I2T) interpretation. However, many generated images still suffer from issues related to perceptual quality and text-image alignment. Given the high cost and inefficiency of manual evaluation, an automatic metric that aligns with human preferences is desirable. To this end, we present EvalMi-50K, a comprehensive dataset and benchmark for evaluating large-multimodal image generation, which features (i) comprehensive tasks, encompassing 2,100 extensive prompts across 20 fine-grained task dimensions, and (ii) large-scale human-preference annotations, including 100K mean-opinion scores (MOSs) and 50K question-answering (QA) pairs annotated on 50,400 images generated from 24 T2I models. Based on EvalMi-50K, we propose LMM4LMM, an LMM-based metric for evaluating large multimodal T2I generation from multiple dimensions including perception, text-image correspondence, and task-specific accuracy. Extensive experimental results show that LMM4LMM achieves state-of-the-art performance on EvalMi-50K, and exhibits strong generalization ability on other AI-generated image evaluation benchmark datasets, manifesting the generality of both the EvalMi-50K dataset and LMM4LMM metric. Both EvalMi-50K and LMM4LMM will be released at https://github.com/IntMeGroup/LMM4LMM.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 71 references that are in the map. First posted on arXiv 11 Apr 2025.

Builds on

48 of 71 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  4. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  5. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  6. 2023LLMGA: Multimodal Large Language Model based Generation Assistant27 Nov
  7. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  4. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  8. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  9. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  10. 2021MUSIQ: Multi-scale Image Quality Transformercited by 1,058
  11. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  12. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
29 more
  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  4. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  5. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  6. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  7. 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 489
  8. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  9. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  10. 2024OmniGen: Unified Image Generationcited by 388
  11. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  12. 2024PixArt-Sigma: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generationcited by 326
  13. 2019Blind Image Quality Assessment Using A Deep Bilinear Convolutional Neural Networkcited by 301
  14. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  15. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  16. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  17. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 285
  18. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  19. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 276
  20. 2024Playground v2.5: Three Insights towards Enhancing Aesthetic Quality in Text-to-Image Generationcited by 249
  21. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  22. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  23. 2024Autoregressive Video Generation without Vector Quantizationcited by 180
  24. 2021No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistencycited by 165
  25. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  26. 2021Blind Quality Assessment for in-the-Wild Images via Hierarchical Feature Fusion and Iterative Mixed Database Trainingcited by 71
  27. 2024EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluationcited by 37
  28. 2024Kandinsky 3: Text-to-Image Synthesis for Multifunctional Generative Frameworkcited by 25
  29. 2024Bridging Different Language Models and Generative Vision Models for Text-to-Image Generationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 14.

24 papers.

Sort:

By area:

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  2. 2025MMIG-Bench: Towards Comprehensive and Explainable Evaluation of Multi-Modal Image Generation Modelscited by 16
  3. 2025LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretationcited by 10
  4. 2025TDVE-Assessor: Benchmarking and Evaluating the Quality of Text-Driven Video Editing with LMMscited by 10
  5. 2026Understanding Reward Hacking in Text-to-Image Reinforcement Learningcited by 9
  6. 2025I2I-Bench: A Comprehensive Benchmark Suite for Image-to-Image Editing Modelscited by 7
  7. 2025TIT-Score: Evaluating Long-Prompt Based Text-to-Image Alignment via Text-to-Image-to-Text Consistencycited by 4
  8. 2026ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Servicescited by 4
  9. 2026Evaluating Image Editing with LLMs: A Comprehensive Benchmark and Intermediate-Layer Probing Approachcited by 3
  10. 2025The Ever-Evolving Science Examcited by 2
  11. 2025Stitch: Training-Free Position Control in Multimodal Diffusion Transformerscited by 2
  12. 2025OmniQuality-R: Advancing Reward Models Through All-Encompassing Quality Assessmentcited by 1
12 more
  1. 2025MagicWand: A Universal Agent for Generation and Evaluation Aligned with User Preferencecited by 1
  2. 2026Q-Bench-Portrait: Benchmarking Multimodal Large Language Models on Portrait Image Quality Perceptioncited by 1
  3. 2026Decoupling Perception and Calibration: Label-Efficient Image Quality Assessment Frameworkcited by 1
  4. 2026ELIQ: A Label-Free Framework for Quality Assessment of Evolving AI-Generated Imagescited by 1
  5. 2026EditRefiner: A Human-Aligned Agentic Framework for Image Editing Refinementcited by 1
  6. 2026On Semiotic-Grounded Interpretive Evaluation of Generative Artcited by 0
  7. 2026DynT2I-Eval: A Dynamic Evaluation Framework for Text-to-Image Modelscited by 0
  8. 2026Do Image Editing Models Understand Lighting?cited by 0
  9. 2026DynEval: Holistic Evaluations of T2I Generative Models in the Wildcited by 0
  10. 2026JoLT: Joint Latent Trajectories for Context-Guided High-Resolution Tiled Generationcited by 0
  11. 2026Bridging Adversarial and Collaborative Learning for AI-Generated Image Quality Assessmentcited by 0
  12. 2026Evaluating the Evaluators: Diagnosing Large Multimodal Models for AI-Generated Image Assessmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MMGenBench: Fully Automatically Evaluating LMMs from the Text-to-Image Generation Perspective
  2. 2025Multimodal Language Models as Text-to-Image Model Evaluators
  3. 2024MMR: Evaluating Reading Ability of Large Multimodal Models
  4. 2025A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation
  5. 2025Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models
  6. 2025MINOS: A Multimodal Evaluation Model for Bidirectional Generation Between Image and Text
  7. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  8. 2025LMM4Edit: Benchmarking and Evaluating Multimodal Image Editing with LMMs