mlsanityData to 14 Oct 2026

MM-Interleaved: Interleaved Image-Text Generative Modeling via Multi-modal Feature Synchronizer

arXiv 2401.10208 ↗First posted 18 Jan 2024cs.CV cs.CLcited by 80

Not on the map. Why?

Cited by 27 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Developing generative models for interleaved image-text data has both research and practical value. It requires models to understand the interleaved sequences and subsequently generate images and text. However, existing attempts are limited by the issue that the fixed number of visual tokens cannot efficiently capture image details, which is particularly problematic in the multi-image scenarios. To address this, this paper presents MM-Interleaved, an end-to-end generative model for interleaved image-text data. It introduces a multi-scale and multi-image feature synchronizer module, allowing direct access to fine-grained image features in the previous context during the generation process. MM-Interleaved is end-to-end pre-trained on both paired and interleaved image-text corpora. It is further enhanced through a supervised fine-tuning phase, wherein the model improves its ability to follow complex multi-modal instructions. Experiments demonstrate the versatility of MM-Interleaved in recognizing visual details following multi-modal instructions and generating consistent images following both textual and visual conditions. Code and models are available at urlhttps://github.com/OpenGVLab/MM-Interleaved.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 99 of its 112 references that are in the map. First posted on arXiv 18 Jan 2024.

Builds on

99 of 112 references.

References · 99

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  12. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
38 more
  1. 2022Classifier-Free Diffusion Guidancecited by 5,847
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,614
  11. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
  12. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  13. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  14. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  15. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  16. 2015VQA: Visual Question Answeringcited by 2,506
  17. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  18. 2019Towards VQA Models That Can Readcited by 1,718
  19. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  20. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  21. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  22. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  23. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  24. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  25. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  26. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  27. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  28. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  29. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  30. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  31. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  32. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  33. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  34. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 691
  35. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  36. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  37. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 625
  38. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615

50 most cited of 99 All 99 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 34; 2025: 32; 2026: 14.

80 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  3. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  4. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 295
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  6. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  7. 2025VisualPRM: An Effective Process Reward Model for Multimodal Reasoningcited by 139
  8. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  9. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 114
  10. 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
  11. 2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
  12. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
38 more
  1. 2025Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunitiescited by 85
  2. 2024SEED-Story: Multimodal Long Story Generation with Large Language Modelcited by 73
  3. 2024OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Textcited by 59
  4. 2024Needle In A Multimodal Haystackcited by 58
  5. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  6. 2024Harmonizing Visual Text Comprehension and Generationcited by 54
  7. 2025Captain Cinema: Towards Short Movie Generationcited by 48
  8. 2024SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Foldingcited by 42
  9. 2024MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversitycited by 38
  10. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  11. 2024DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Modelscited by 36
  12. 2025Benchmarking Retrieval-Augmented Multimodal Generation for Document Question Answeringcited by 36
  13. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  14. 2024DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perceptioncited by 32
  15. 2024What Factors Affect Multi-Modal In-Context Learning? An In-Depth Explorationcited by 31
  16. 2024Holistic Evaluation for Interleaved Text-and-Image Generationcited by 28
  17. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  18. 2026InternVL-U: Democratizing Unified Multimodal Models for Understanding, Reasoning, Generation and Editingcited by 28
  19. 2024CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generationcited by 23
  20. 2024The Evolution of Multimodal Model Architecturescited by 21
  21. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 21
  22. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  23. 2025Docopilot: Improving Multimodal Models for Document-Level Understandingcited by 19
  24. 2024Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challengescited by 18
  25. 2025VideoAuteur: Towards Long Narrative Video Generationcited by 18
  26. 2025Generative AI for Cel-Animation: A Surveycited by 17
  27. 2025Watermarking Autoregressive Image Generationcited by 16
  28. 2024Unified Generative and Discriminative Training for Multi-modal Large Language Modelscited by 15
  29. 2025Cross-Task Generalization Between Understanding and Generation in Unified Vision-Language Models: A Controlled Studycited by 15
  30. 2025Pre-Trained Policy Discriminators are General Reward Modelscited by 15
  31. 2025GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoningcited by 14
  32. 2025Unified Dense Prediction of Video Diffusioncited by 13
  33. 2026LLaMo: Scaling Pretrained Language Models for Unified Motion Understanding and Generation with Continuous Autoregressive Tokenscited by 11
  34. 2026UniM: A Unified Any-to-Any Interleaved Multimodal Benchmarkcited by 11
  35. 2024Vision Model Pre-training on Interleaved Image-Text Data via Latent Compression Learningcited by 9
  36. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  37. 2024Knowledge-Aware Reasoning over Multimodal Semi-structured Tablescited by 9
  38. 2024EMMA: Efficient Visual Alignment in Multi-Modal LLMscited by 8

50 most cited of 80 All 80 (JSON)