mlsanityData to 7 Oct 2026

Emu: Generative Pretraining in Multimodality

arXiv 2307.05222 ↗First posted 11 Jul 2023cs.CVcited by 156

Not on the map. Why?

Cited by 60 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present Emu, a Transformer-based multimodal foundation model, which can seamlessly generate images and texts in multimodal context. This omnivore model can take in any single-modality or multimodal data input indiscriminately (e.g., interleaved image, text and video) through a one-model-for-all autoregressive training process. First, visual signals are encoded into embeddings, and together with text tokens form an interleaved input sequence. Emu is then end-to-end trained with a unified objective of classifying the next text token or regressing the next visual embedding in the multimodal sequence. This versatile multimodality empowers the exploration of diverse pretraining data sources at scale, such as videos with interleaved frames and text, webpages with interleaved images and text, as well as web-scale image-text pairs and video-text pairs. Emu can serve as a generalist multimodal interface for both image-to-text and text-to-image tasks, and supports in-context image and text generation. Across a broad range of zero-shot/few-shot tasks including image captioning, visual question answering, video question answering and text-to-image generation, Emu demonstrates superb performance compared to state-of-the-art large multimodal models. Extended capabilities such as multimodal assistants via instruction tuning are also demonstrated with impressive performance.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Builds on 76 of its 90 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

76 of 90 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  11. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  12. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
38 more
  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  13. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  14. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  15. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  16. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  17. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  18. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  19. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  20. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  21. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  22. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  23. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  24. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  25. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  26. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  27. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  28. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  29. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  30. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  31. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  32. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  33. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  34. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  35. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  36. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  37. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  38. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 85; 2025: 30; 2026: 11.

156 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,005
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  6. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 760
  7. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  8. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 591
  12. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
38 more
  1. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
  2. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  3. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  4. 2024SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generationcited by 328
  5. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  6. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  7. 2024Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesiscited by 297
  8. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 296
  9. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 295
  10. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  11. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  12. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 241
  13. 2025Transfer between Modalities with MetaQueriescited by 231
  14. 2024Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMscited by 224
  15. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  16. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 213
  17. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 188
  18. 2023Making LLaMA SEE and Draw with SEED Tokenizercited by 184
  19. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 182
  20. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  21. 2025FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Drivingcited by 165
  22. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 162
  23. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 162
  24. 2025UniTok: A Unified Tokenizer for Visual Generation and Understandingcited by 148
  25. 2023Planting a SEED of Vision in Large Language Modelcited by 140
  26. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  27. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  28. 2024VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Taskscited by 136
  29. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  30. 2024LMFusion: Adapting Pretrained Language Models for Multimodal Generationcited by 120
  31. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  32. 2023Kosmos-G: Generating Images in Context with Multimodal Large Language Modelscited by 105
  33. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 100
  34. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  35. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  36. 2023CapsFusion: Rethinking Image-Text Data at Scalecited by 90
  37. 2024Unveiling Encoder-Free Vision-Language Modelscited by 90
  38. 2023Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenizationcited by 87

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Emu3: Next-Token Prediction is All You Need
  2. 2023Generative Multimodal Models are In-Context Learners
  3. 2024MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data
  4. 2025Emu3.5: Native Multimodal Models are World Learners
  5. 2024VIMI: Grounding Video Generation through Multi-modal Instruction
  6. 2026Beyond Language Modeling: An Exploration of Multimodal Pretraining
  7. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystack
  8. 2024Towards Multi-Task Multi-Modal Models: A Video Generative Perspective