# MMEB-V3: Measuring the Performance Gaps of Omni-Modality Embedding Models

arXiv: https://arxiv.org/abs/2604.23321 · HTML: https://mlsanity.com/paper/2604.23321/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Apr 2026.

## Built on by (most cited first, 8 of 8)

- [WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report](https://mlsanity.com/paper/2608.24053/) (2026)
- [MVEB: Massive Video Embedding Benchmark](https://mlsanity.com/paper/2606.14958/) (2026)
- [CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoring](https://mlsanity.com/paper/2606.11700/) (2026)
- [MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models](https://mlsanity.com/paper/2609.08663/) (2026)
- [Ovis-Embedding: Pushing the Frontiers of Universal Omni-Modal Embeddings](https://mlsanity.com/paper/2609.25165/) (2026)
- [World Embedding Benchmark](https://mlsanity.com/paper/2610.03632/) (2026)
- [MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface](https://mlsanity.com/paper/2610.11316/) (2026)
- [What Really Works for Universal Multimodal Embeddings? A Controlled Study of Contrastive Training Strategies](https://mlsanity.com/paper/2610.15035/) (2026)

## Abstract (arXiv)

Multimodal embedding models aim to map heterogeneous inputs, such as text, images, videos, and audio, into a shared semantic space. However, existing methods and benchmarks remain largely limited to partial modality coverage, making it difficult to systematically evaluate full-modality representation learning. In this work, we take a step toward the full-modality setting. We introduce MMEB-V3, a comprehensive benchmark that evaluates embeddings across text, image, video, audio, as well as agent-centric scenarios. To enable more fine-grained diagnosis, we further construct OmniSET (Omni-modality Semantic Equivalence Tuples), where semantically equivalent instances are represented across modalities, allowing us to disentangle semantic similarity from modality effects. Through experiments on MMEB-V3, we conduct a systematic analysis of full-modality embeddings and identify three key findings: (1) models often fail to retrieve the intended target modality; (2) cross-modal retrieval is highly asymmetric and dominated by query-modality bias; and (3) instruction-induced shifts are either insufficient or misaligned with the target modality, and therefore do not reliably improve retrieval. These results indicate that current multimodal embeddings are not yet capable of reliably enforcing modality constraints specified by instructions, and consequently fail to exhibit consistent modality-aware retrieval behavior. We hope MMEB-V3 provides a useful benchmark for understanding and diagnosing these limitations, and for guiding future research on full-modality embeddings.