mlsanityData to 14 Oct 2026

Omni-Embed-Mini: Binding Modalities Without Forgetting via Dense Distillation

arXiv 2610.02148 ↗First posted 1 Oct 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Extending a text embedding model to new modalities typically degrades text retrieval quality, and existing omni-modal embedders compensate with multi-billion parameters. We present Omni-Embed-Mini, a 0.9B-parameter model that maps text, speech, audio, images, video, and visually-rich documents into a single shared cosine space without updating any text-side parameter. Our key insight is that the teacher signal requires no separate embedding model: each media sample is paired with a dense cascaded caption, and the teacher target is simply the frozen backbone's own embedding of that caption. Because teacher and student share the same backbone weights, they inhabit byte-identical geometry, and lightweight projectors plus phased LoRA adapters on the modality encoders suffice for alignment. Training combines a Matryoshka SigLIP contrastive loss with an online hybrid hard-negative miner whose negatives sharpen as the encoder improves. The recipe carries over to a 2.3B variant by swapping in a native vision-language backbone. Omni-Embed-Mini-0.9B keeps its text weights bit-identical to the backbone, so training cannot regress text retrieval (49.57 nDCG@10 on MTEB-v2 BEIR-8), while extending it to five additional modalities, and is ~2.7x to 9.5x smaller than every open omni embedder we compare against. The 2.3B variant is competitive with the closed gemini-embedding-2, edging ahead of it on the overall-modality average. Models, code, data and evaluation harness are on our project page: https://omniembed.cvmbzuai.com

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

32 of 54 references.

References · 32

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2017Billion-scale similarity search with GPUscited by 2,497
  5. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  6. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  7. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  8. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 849
  9. 2017Dense-Captioning Events in Videoscited by 793
  10. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  11. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  12. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 578
20 more
  1. 2019Clotho: An Audio Captioning Datasetcited by 358
  2. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  3. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  4. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 245
  5. 2024Generative Representational Instruction Tuningcited by 233
  6. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  7. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 226
  8. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 139
  9. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
  10. 2023Natural Language Supervision for General-Purpose Audio Representationscited by 61
  11. 2024Scaling up masked audio encoder learning for general audio classificationcited by 44
  12. 2025Think Then Embed: Generative Context Improves Multimodal Embeddingcited by 41
  13. 2025RzenEmbed: Towards Comprehensive Multimodal Retrievalcited by 29
  14. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 26
  15. 2025Scaling Language-Centric Omnimodal Representation Learningcited by 22
  16. 2023HeySQuAD: A Spoken Question Answering Datasetcited by 16
  17. 2026e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddingscited by 13
  18. 2023GSQA: An End-to-End Model for Generative Spoken Question Answeringcited by 9
  19. 2026WeMM-Embedding: WeChat Multi-Modal Embedding Technical Reportcited by 8
  20. 2026BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMscited by 5

Built on by

Not cited yet.