mlsanityData to 7 Oct 2026

SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

arXiv 2412.09604 ↗First posted 12 Dec 2024cs.CVcited by 42

Not on the map. Why?

Abstract (arXiv)

The remarkable success of Large Language Models (LLMs) has extended to the multimodal domain, achieving outstanding performance in image understanding and generation. Recent efforts to develop unified Multimodal Large Language Models (MLLMs) that integrate these capabilities have shown promising results. However, existing approaches often involve complex designs in model architecture or training pipeline, increasing the difficulty of model training and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful encoder-free MLLM capable of both image understanding and generation. To address challenges identified in existing encoder-free unified MLLMs, we introduce the token folding mechanism and the vision-expert-based progressive alignment pretraining strategy, which effectively support high-resolution image understanding while reducing training complexity. After being trained on large-scale mixed image-text data with a unified next-token prediction objective, SynerGen-VL achieves or surpasses the performance of existing encoder-free unified MLLMs with comparable or smaller parameter sizes, and narrows the gap with task-specific state-of-the-art models, highlighting a promising path toward future unified MLLMs. Our code and models shall be released.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 85 of its 104 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

85 of 104 references.

References · 85

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
38 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2023Qwen Technical Reportcited by 3,273
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  9. 2019Towards VQA Models That Can Readcited by 1,696
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  12. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  13. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  14. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  15. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  16. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  17. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  18. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 946
  19. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  20. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  21. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 854
  22. 2016A Diagram Is Worth A Dozen Imagescited by 793
  23. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  24. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  25. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  26. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  27. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  28. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  29. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  30. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  31. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  32. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  33. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 443
  34. 2021InfographicVQAcited by 430
  35. 2021OCR-free Document Understanding Transformercited by 400
  36. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  37. 2024InternLM2 Technical Reportcited by 391
  38. 2024OmniGen: Unified Image Generationcited by 388

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 12.

42 papers.

Sort:

By area:

  1. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  2. 2025WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generationcited by 196
  3. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  4. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  5. 2025Envisioning Beyond the Pixels: Benchmarking Reasoning-Informed Visual Editingcited by 97
  6. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 81
  7. 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
  8. 2025OneCAT: Decoder-Only Auto-Regressive Model for Unified Understanding and Generationcited by 50
  9. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 45
  10. 2025UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editingcited by 43
  11. 2025Skywork UniPic 2.0: Building Kontext Model with Online RL for Unified Multimodal Modelcited by 38
  12. 2025ImageGen-CoT: Enhancing Text-to-Image In-context Learning with Chain-of-Thought Reasoningcited by 37
30 more
  1. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  2. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  3. 2025From Forecasting to Planning: Policy World Model for Collaborative State-Action Predictioncited by 29
  4. 2025UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learningcited by 23
  5. 2025UniFlow: A Unified Pixel Flow Tokenizer for Visual Understanding and Generationcited by 23
  6. 2025SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generationcited by 21
  7. 2025GenExam: A Multidisciplinary Text-to-Image Examcited by 19
  8. 2025Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Modelscited by 18
  9. 2025HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generationcited by 17
  10. 2025Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Spacecited by 16
  11. 2025Co-Reinforcement Learning for Unified Multimodal Understanding and Generationcited by 15
  12. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13
  13. 2025UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generationcited by 11
  14. 2025AdaRD-key: Adaptive Relevance-Diversity Keyframe Sampling for Long-form Video understandingcited by 9
  15. 2025Does Understanding Inform Generation in Unified Multimodal Models? From Analysis to Path Forwardcited by 9
  16. 2025Parameter-Inverted Image Pyramid Networks for Visual Perception and Multimodal Understandingcited by 8
  17. 2026GRADE: Benchmarking Discipline-Informed Reasoning in Image Editingcited by 5
  18. 2026Towards Holistic Modeling for Video Frame Interpolation with Auto-regressive Diffusion Transformerscited by 4
  19. 2026UniX: Unifying Autoregression and Diffusion for Chest X-Ray Understanding and Generationcited by 3
  20. 2025UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasetscited by 1
  21. 2025ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interactioncited by 1
  22. 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
  23. 2025Conjuring Positive Pairs for Efficient Unification of Representation Learning and Image Synthesiscited by 0
  24. 2026SSVP: Synergistic Semantic-Visual Prompting for Industrial Zero-Shot Anomaly Detectioncited by 0
  25. 2026Token by Token, Compromised: Backdoor Vulnerabilities in Unified Autoregressive Modelscited by 0
  26. 2026Learning from Semantic Dictionaries: Discriminative Codebook Contrastive Learning for Unified Visual Representation and Generationcited by 0
  27. 2026Twins: Learn to Predict Unified Representations with Focal Losscited by 0
  28. 2026C3-UniMM: Causal Cycle-Consistent Unified Multimodal Modeling via Super Alignment and Shared Decoding Spacecited by 0
  29. 2026How Far Are We from Removing the Visual Encoder? Scaling Laws for Encoder-Free Multimodal Pretrainingcited by 0
  30. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  2. 2024MUSE-VL: Modeling Unified VLM through Semantic Discrete Encoding
  3. 2026UVLM: A Universal Vision-Language Model Loader for Reproducible Multimodal Benchmarking
  4. 2025Visual Representation Alignment for Multimodal Large Language Models
  5. 2026Argus-Unified: Towards A Compact and Economical Unified Model for Image Understanding and Generation
  6. 2024The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models
  7. 2025VUGEN: Visual Understanding priors for GENeration
  8. 2025A Training-Free, Task-Agnostic Framework for Enhancing MLLM Performance on High-Resolution Images