mlsanityData to 7 Oct 2026

JourneyDB: A Benchmark for Generative Image Understanding

arXiv 2307.00716 ↗First posted 3 Jul 2023cs.CVcited by 203

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

While recent advancements in vision-language models have had a transformative impact on multi-modal comprehension, the extent to which these models possess the ability to comprehend generated images remains uncertain. Synthetic images, in comparison to real data, encompass a higher level of diversity in terms of both content and style, thereby presenting significant challenges for the models to fully grasp. In light of this challenge, we introduce a comprehensive dataset, referred to as JourneyDB, that caters to the domain of generative images within the context of multi-modal visual understanding. Our meticulously curated dataset comprises 4 million distinct and high-quality generated images, each paired with the corresponding text prompts that were employed in their creation. Furthermore, we additionally introduce an external subset with results of another 22 text-to-image generative models, which makes JourneyDB a comprehensive benchmark for evaluating the comprehension of generated images. On our dataset, we have devised four benchmarks to assess the performance of generated image comprehension in relation to both content and style interpretation. These benchmarks encompass prompt inversion, style retrieval, image captioning, and visual question answering. Lastly, we evaluate the performance of state-of-the-art multi-modal models when applied to the JourneyDB dataset, providing a comprehensive analysis of their strengths and limitations in comprehending generated content. We anticipate that the proposed dataset and benchmarks will facilitate further research in the field of generative content understanding. The dataset is publicly available at https://journeydb.github.io.

Key facts. Cited by 203 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 51 of its 69 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

51 of 69 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  3. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  10. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  11. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  12. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 847
  13. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  14. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  15. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  16. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  17. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  18. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  19. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  20. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  21. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  22. 2015Visual7W: Grounded Question Answering in Imagescited by 315
  23. 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
  24. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  25. 2023Improving CLIP Training with Language Rewritescited by 216
  26. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  27. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  28. 2022Can large language models reason about medical questions?cited by 194
  29. 2015DRAW: A Recurrent Neural Network For Image Generationcited by 190
  30. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  31. 2015Generating Images from Captions with Attentioncited by 135
  32. 2021LAFITE: Towards Language-Free Training for Text-to-Image Generationcited by 126
  33. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  34. 2021FuseDream: Training-Free Text-to-Image Generation with Improved CLIP+GAN Space Optimizationcited by 71
  35. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  36. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  37. 2022Controllable 3D Face Synthesis with Conditional Generative Occupancy Fieldscited by 37
  38. 2021Inverting Generative Adversarial Renderer for Face Reconstructioncited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 68; 2025: 80; 2026: 48.

203 papers.

Sort:

By area:

  1. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  2. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  3. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  4. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 568
  5. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  6. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 455
  7. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  8. 2024OmniGen: Unified Image Generationcited by 388
  9. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  10. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  11. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 286
  12. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
38 more
  1. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 241
  2. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 225
  3. 2024VideoPhy: Evaluating Physical Commonsense for Video Generationcited by 215
  4. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  5. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 196
  6. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  7. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  8. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  9. 2024Open-MAGVIT2: An Open-Source Project Toward Democratizing Auto-regressive Visual Generationcited by 153
  10. 2023SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillationcited by 109
  11. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  12. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  13. 2024SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editingcited by 97
  14. 2024PeRFlow: Piecewise Rectified Flow as Universal Plug-and-Play Acceleratorcited by 95
  15. 2025Echo-4o: Harnessing the Power of GPT-4o Synthetic Images for Improved Image Generationcited by 95
  16. 2024MovieDreamer: Hierarchical Generation for Coherent Long Visual Sequencecited by 85
  17. 2025SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RLcited by 85
  18. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  19. 2025PixNerd: Pixel Neural Field Diffusioncited by 75
  20. 2024ChronoMagic-Bench: A Benchmark for Metamorphic Evaluation of Text-to-Time-lapse Video Generationcited by 74
  21. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  22. 2025Thinking with Generated Imagescited by 71
  23. 2025Ovis-U1 Technical Reportcited by 65
  24. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  25. 2024StyleShot: A Snapshot on Any Stylecited by 60
  26. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  27. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  28. 2025BLIP3o-NEXT: Next Frontier of Native Image Generationcited by 50
  29. 2026Scaling Text-to-Image Diffusion Transformers with Representation Autoencoderscited by 49
  30. 2025Democratizing Text-to-Image Masked Generative Models with Compact Text-Aware One-Dimensional Tokenscited by 48
  31. 2025Reconstruction Alignment Improves Unified Multimodal Modelscited by 48
  32. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  33. 2024SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its Teachercited by 45
  34. 2024MonoFormer: One Transformer for Both Diffusion and Autoregressioncited by 45
  35. 2024Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Headscited by 44
  36. 2024Scalable Image Tokenization with Index Backpropagation Quantizationcited by 44
  37. 2025Vision as a Dialect: Unifying Visual Understanding and Generation via Text-Aligned Representationscited by 44
  38. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43

50 most cited of 203 All 203 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing Through Words: Controlling Visual Retrieval Quality with Language Models
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension
  3. 2026Generative Embedding Benchmark: How Much Information Survives in a Dense Embedding?
  4. 2025IF-Bench: Benchmarking and Enhancing MLLMs for Infrared Images with Generative Visual Prompting
  5. 2024Harmonizing Visual Text Comprehension and Generation
  6. 2024OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation
  7. 2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedback
  8. 2025UniPercept: Towards Unified Perceptual-Level Image Understanding across Aesthetics, Quality, Structure, and Texture