mlsanityData to 14 Oct 2026

Multi-Modal Generative Embedding Model

arXiv 2405.19333 ↗First posted 29 May 2024cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Most multi-modal tasks can be formulated into problems of either generation or embedding. Existing models usually tackle these two types of problems by decoupling language modules into a text decoder for generation, and a text encoder for embedding. To explore the minimalism of multi-modal paradigms, we attempt to achieve only one model per modality in this work. We propose a Multi-Modal Generative Embedding Model (MM-GEM), whereby the generative and embedding objectives are encapsulated in one Large Language Model. We also propose a PoolAggregator to boost efficiency and enable the ability of fine-grained embedding and generation. A surprising finding is that these two objectives do not significantly conflict with each other. For example, MM-GEM instantiated from ViT-Large and TinyLlama shows competitive performance on benchmarks for multimodal embedding models such as cross-modal retrieval and zero-shot classification, while has good ability of image captioning. Additionally, MM-GEM can seamlessly execute region-level image caption generation and retrieval tasks. Besides, the advanced text model in MM-GEM brings over 5% improvement in Recall@1 for long text and image retrieval.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

38 of 52 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Segment Anythingcited by 8,234
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
26 more
  1. 2013Describing Textures in the Wildcited by 2,125
  2. 2013Fine-Grained Visual Classification of Aircraftcited by 1,726
  3. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  4. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  7. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  9. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 797
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  11. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  12. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  13. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 490
  14. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  15. 2018nocaps: novel object captioning at scalecited by 444
  16. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  17. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  18. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  19. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  20. 2024Generative Representational Instruction Tuningcited by 233
  21. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  22. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  23. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  24. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 108
  25. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-trainingcited by 67
  26. 2021Unifying Multimodal Transformer for Bi-directional Image and Text Generationcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understandingcited by 25
  2. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 22
  3. 2025CoLLM: A Large Language Model for Composed Image Retrievalcited by 21
  4. 2025VLog: Video-Language Models by Generative Retrieval of Narration Vocabularycited by 7
  5. 2026CREM: Compression-Driven Representation Enhancement for Multimodal Retrieval and Comprehensioncited by 4
  6. 2025You Only Forward Once: An Efficient Compositional Judging Paradigmcited by 1
  7. 2026On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understandingcited by 1