mlsanityData to 7 Oct 2026

SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation

arXiv 2404.14396 ↗First posted 22 Apr 2024cs.CVcited by 328

Not on the map. Why?

Cited by 75 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid evolution of multimodal foundation model has demonstrated significant progresses in vision-language understanding and generation, e.g., our previous work SEED-LLaMA. However, there remains a gap between its capability and the real-world applicability, primarily due to the model's limited capacity to effectively respond to various user instructions and interact with diverse visual data. In this work, we focus on bridging this gap through integrating two enhanced features: (1) comprehending images of arbitrary sizes and ratios, and (2) enabling multi-granularity image generation. We present a unified and versatile foundation model, namely, SEED-X, which is able to model multi-granularity visual semantics for comprehension and generation tasks. Besides the competitive results on public benchmarks, SEED-X demonstrates its effectiveness in handling real-world applications across various domains after instruction tuning. We hope that our work will inspire future research into what can be achieved by versatile multimodal foundation models in real-world applications. The models, codes, and datasets are released in https://github.com/AILab-CVC/SEED-X.

Key facts. Cited by 328 papers in the map (data through 7 Oct 2026). Builds on 56 of its 70 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

56 of 70 references.

References · 56

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Segment Anythingcited by 8,158
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  12. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  14. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  15. 2016A Diagram Is Worth A Dozen Imagescited by 793
  16. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  17. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  18. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  19. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 560
  20. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  21. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  22. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 338
  23. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  24. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  25. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  26. 2021VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalizationcited by 287
  27. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  28. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  29. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 264
  30. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  31. 2023JourneyDB: A Benchmark for Generative Image Understandingcited by 203
  32. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  33. 2016Visual Storytellingcited by 176
  34. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Modelcited by 172
  35. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  36. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  37. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 143
  38. 2023Planting a SEED of Vision in Large Language Modelcited by 140

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 57; 2025: 169; 2026: 102.

328 papers.

Sort:

By area:

  1. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  2. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  3. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  4. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  5. 2024OmniGen: Unified Image Generationcited by 388
  6. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  7. 2025LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through Two-Stage Rule-Based RLcited by 244
  8. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  9. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  10. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 235
  11. 2025Transfer between Modalities with MetaQueriescited by 231
  12. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
38 more
  1. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 193
  2. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  3. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  4. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  5. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  6. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  7. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  8. 2024DreamBench++: A Human-Aligned Benchmark for Personalized Image Generationcited by 131
  9. 2025Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Stepcited by 130
  10. 2024HART: Efficient Visual Generation with Hybrid Autoregressive Transformercited by 128
  11. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  12. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  13. 2025HealthGPT: A Medical Large Vision-Language Model for Unifying Comprehension and Generation via Heterogeneous Knowledge Adaptationcited by 112
  14. 2025Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generationcited by 108
  15. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 105
  16. 2025GoT: Unleashing Reasoning Capability of Multimodal Large Language Model for Visual Generation and Editingcited by 103
  17. 2024UniReal: Universal Image Generation and Editing via Learning Real-world Dynamicscited by 100
  18. 2025UP-VLA: A Unified Understanding and Prediction Model for Embodied Agentcited by 99
  19. 2024SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editingcited by 97
  20. 2025ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generationcited by 94
  21. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentationcited by 86
  22. 2025UniVideo: Unified Understanding, Generation, and Editing for Videoscited by 85
  23. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 83
  24. 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
  25. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 70
  26. 2025UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generationcited by 64
  27. 2024Liquid: Language Models are Scalable and Unified Multi-modal Generatorscited by 62
  28. 2024Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoningcited by 57
  29. 2025GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generationcited by 57
  30. 2024ACE: All-round Creator and Editor Following Instructions via Diffusion Transformercited by 56
  31. 2025UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokenscited by 55
  32. 2025NextStep-1: Toward Autoregressive Image Generation with Continuous Tokens at Scalecited by 55
  33. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  34. 2024Reconstructive Visual Instruction Tuningcited by 54
  35. 2025ILLUME+: Illuminating Unified MLLM with Dual Visual Tokenization and Diffusion Refinementcited by 53
  36. 2025Muddit: Liberating Generation Beyond Text-to-Image with a Unified Discrete Diffusion Modelcited by 52
  37. 2025Uni-cot: Towards Unified Chain-of-Thought Reasoning Across Text and Visioncited by 52
  38. 2025GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learningcited by 51

50 most cited of 328 All 328 (JSON)