Data to 14 Oct 2026

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

arXiv 2406.04292 ↗First posted 6 Jun 2024cs.IR cs.CL cs.CVcited by 67

Not on the map. Why?

Abstract (arXiv)

Multi-modal retrieval becomes increasingly popular in practice. However, the existing retrievers are mostly text-oriented, which lack the capability to process visual information. Despite the presence of vision-language models like CLIP, the current methods are severely limited in representing the text-only and image-only data. In this work, we present a new embedding model VISTA for universal multi-modal retrieval. Our work brings forth threefold technical contributions. Firstly, we introduce a flexible architecture which extends a powerful text encoder with the image understanding capability by introducing visual token embeddings. Secondly, we develop two data generation strategies, which bring high-quality composed image-text to facilitate the training of the embedding model. Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data. In our experiments, VISTA achieves superior performances across a variety of multi-modal retrieval tasks in both zero-shot and supervised settings. Our model, data, and source code are available at https://github.com/FlagOpen/FlagEmbedding.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Builds on 33 of its 43 references that are in the map. First posted on arXiv 6 Jun 2024.

Builds on

33 of 43 references.

References · 33

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2022Training language models to follow instructions with human feedbackcited by 15,971
2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611

21 more

2017Billion-scale similarity search with GPUscited by 2,497
2021Improving language models by retrieving from trillions of tokenscited by 1,293
2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
2021Large Dual Encoders Are Generalizable Retrieverscited by 456
2022Text and Code Embeddings by Contrastive Pre-Trainingcited by 361
2022Scaling Language-Image Pre-training via Maskingcited by 317
2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
2021Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Modelscited by 251
2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 236
2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
2023UniIR: Training and Benchmarking Universal Multimodal Information Retrieverscited by 166
2022Autoregressive Search Engines: Generating Substrings as Document Identifierscited by 113
2023Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entitiescited by 113
2022Universal Vision-Language Dense Retrieval: Learning A Unified Representation Space for Multi-Modal Retrievalcited by 58
2023Bi-directional Training for Composed Image Retrieval via Text Prompt Learningcited by 43
2023End-to-end Knowledge Retrieval with Multi-modal Queriescited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 36; 2026: 24.

67 papers.

Sort:

By area:

2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 170
2024E5-V: Universal Embeddings with Multimodal Large Language Modelscited by 126
2025mmE5: Improving Multimodal Multilingual Embeddings via High-quality Synthetic Datacited by 49
2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
2025ModernVBERT: Towards Smaller Visual Document Retrieverscited by 21
2024CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrievalcited by 19
2025A Comprehensive Survey on Composed Image Retrievalcited by 17
2025O1 Embedder: Let Retrievers Think Before Actioncited by 15
2025MIEB: Massive Image Embedding Benchmarkcited by 15
2026According to Me: Long-Term Personalized Referential Memory QAcited by 14

38 more

2025Uni-Retrieval: A Multi-Style Retrieval Framework for STEM's Educationcited by 13
2025Benchmarking Retrieval-Augmented Generation in Multi-Modal Contextscited by 12
2025UniECS: Unified Multimodal E-Commerce Search Framework with Gated Cross-modal Fusioncited by 10
2025Memory-QA: Answering Recall Questions Based on Multimodal Memoriescited by 10
2025SAIL-Embedding Technical Report: Omni-modal Embedding Foundation Modelcited by 10
2025Any Information Is Just Worth One Single Screenshot: Unifying Search With Visualized Information Retrievalcited by 8
2025Towards Temporal-Aware Multi-Modal Retrieval Augmented Generation in Financecited by 8
2026Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrievalcited by 8
2026WeMM-Embedding: WeChat Multi-Modal Embedding Technical Reportcited by 8
2025Generalized Contrastive Learning for Universal Multimodal Retrievalcited by 7
2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
2026CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embeddingcited by 7
2024Unified Multimodal Interleaved Document Representation for Retrievalcited by 6
2024GeoCoder: Solving Geometry Problems by Generating Modular Code through Vision-Language Modelscited by 5
2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddingscited by 5
2025Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrievalcited by 5
2025From Generator to Embedder: Harnessing Innate Abilities of Multimodal LLMs via Building Zero-Shot Discriminative Embedding Modelcited by 5
2026jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towerscited by 5
2025From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videoscited by 4
2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processingcited by 4
2025A Survey of Deep Learning for Geometry Problem Solvingcited by 4
2024MIRe: Enhancing Multimodal Queries Representation via Fusion-Free Modality Interaction for Multimodal Retrievalcited by 3
2025FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokenscited by 3
2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledgecited by 3
2025InfiR : Crafting Effective Small Language Models and Multimodal Small Language Models in Reasoningcited by 2
2025Towards Text-Image Interleaved Retrievalcited by 2
2025Composed Multi-modal Retrieval: A Survey of Approaches and Applicationscited by 2
2025HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrievalcited by 2
2025From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEMcited by 2
2025Roles of MLLMs in Visually Rich Document Retrieval for RAG: A Surveycited by 2
2026Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Spacecited by 2
2026Pailitao-VL: Unified Embedding and Reranker for Real-Time Multi-Modal Industrial Searchcited by 2
2026NanoVDR: Distilling a 2B Vision-Language Retriever into a 70M Text-Only Encoder for Visual Document Retrievalcited by 2
2025Boosting Text-to-Chart Retrieval through Training with Synthesized Semantic Insightscited by 1
2025SORCE: Small Object Retrieval in Complex Environmentscited by 1
2025Exploring a Unified Vision-Centric Contrastive Alternatives on Multi-Modal Web Documentscited by 1
2025Windsock is Dancing: Adaptive Multimodal Retrieval-Augmented Generationcited by 1
2025Bridging the Modality Gap by Similarity Standardization with Pseudo-Positive Samplescited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way