# Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video

arXiv: https://arxiv.org/abs/2510.03458 · HTML: https://mlsanity.com/paper/2510.03458/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 27 references that are in the map.
- First posted on arXiv 3 Oct 2025.

## Builds on (most cited first, 11 of 11)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Multilingual E5 Text Embeddings: A Technical Report](https://mlsanity.com/paper/2402.05672/) (2024)
- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [Jasper and Stella: distillation of SOTA embedding models](https://mlsanity.com/paper/2412.19048/) (2024)
- [NV-Retriever: Improving text embedding models with effective hard-negative mining](https://mlsanity.com/paper/2407.15831/) (2024)
- [Tevatron 2.0: Unified Document Retrieval Toolkit across Scale, Language, and Modality](https://mlsanity.com/paper/2505.02466/) (2025)
- [Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model](https://mlsanity.com/paper/2507.05513/) (2025)
- [Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks](https://mlsanity.com/paper/2506.21182/) (2025)

## Built on by (most cited first, 16 of 16)

- [e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings](https://mlsanity.com/paper/2601.03666/) (2026)
- [WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report](https://mlsanity.com/paper/2608.24053/) (2026)
- [Discovering Functionally Selective Brain Regions with a Deep Topographic Multimodal Model](https://mlsanity.com/paper/2606.09770/) (2026)
- [SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise](https://mlsanity.com/paper/2602.12783/) (2026)
- [Multimodal Data Curation Through Ranked Retrieval](https://mlsanity.com/paper/2605.01163/) (2026)
- [Omni-Interactive Universal Embedder](https://mlsanity.com/paper/2608.27044/) (2026)
- [DSE-VTG: Dual-Side Enhancement for Training-Free Video Temporal Grounding](https://mlsanity.com/paper/2609.08850/) (2026)
- [Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings](https://mlsanity.com/paper/2609.25165/) (2026)
- [MultiVENT-Raw: A Benchmark for Retrieval and Reasoning over Raw Videos](https://mlsanity.com/paper/2609.28437/) (2026)
- [PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?](https://mlsanity.com/paper/2609.34314/) (2026)
- [How To Train Your World Model: Fine-tuning vs RAG for LM-based World Modeling](https://mlsanity.com/paper/2610.02542/) (2026)
- [World Embedding Benchmark](https://mlsanity.com/paper/2610.03632/) (2026)
- [RepICL: Reusable In-Context Prediction Across Heterogeneous Representation Spaces](https://mlsanity.com/paper/2610.05852/) (2026)
- [Syn-Omni: Structured Specialization and Progressive Collaboration for Omnimodal Embeddings](https://mlsanity.com/paper/2610.12256/) (2026)
- [Omni-Modal LLMs Know More Than They Say](https://mlsanity.com/paper/2610.16135/) (2026)
- [Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution](https://mlsanity.com/paper/2610.19495/) (2026)

## Abstract (arXiv)

We present Omni-Embed-Nemotron, a unified multimodal retrieval embedding model developed to handle the increasing complexity of real-world information needs. While Retrieval-Augmented Generation (RAG) has significantly advanced language models by incorporating external knowledge, existing text-based retrievers rely on clean, structured input and struggle with the visually and semantically rich content found in real-world documents such as PDFs, slides, or videos. Recent work such as ColPali has shown that preserving document layout using image-based representations can improve retrieval quality. Building on this, and inspired by the capabilities of recent multimodal models such as Qwen2.5-Omni, we extend retrieval beyond text and images to also support audio and video modalities. Omni-Embed-Nemotron enables both cross-modal (e.g., text - video) and joint-modal (e.g., text - video+audio) retrieval using a single model. We describe the architecture, training setup, and evaluation results of Omni-Embed-Nemotron, and demonstrate its effectiveness in text, image, and video retrieval.