mlsanityData to 14 Oct 2026

Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

arXiv 2607.18666 ↗First posted 21 Jul 2026cs.CL cs.SDcited by 3

Not on the map. Why?

Abstract (arXiv)

A single embedding space that covers text, images, video, and audio lets one index serve every query a user can pose. Embedding models built on vision-language backbones now lead text/image/video retrieval benchmarks but lack audio entirely, while audio-text retrieval is led by specialist systems that serve no other modality. We present the Fusion Embedding family, which adds audio to a frozen vision-language embedding base whose parameters are never updated: generation 1 (fusion-embedding-1) trains only a 16.4M-parameter connector between a frozen audio tower and the frozen base, and generation 2 (fusion-embedding-2) adds modality-gated deep adapters (44.2M parameters) whose branch never executes on text, image, or video inputs: their outputs are bit-for-bit those of the released base, verified after every training run. Because the base already binds text, images, and video, aligning audio to text alone makes audio-image retrieval emerge, with zero paired audio-visual training data. Alongside the recipe we map its design space with controlled negative results (rewriting training captions with an LLM, substituting a leaderboard-stronger audio tower, and widening the connector each reduce retrieval) and with training-protocol findings that we expect to transfer to any frozen decoder-LM embedding backbone. Both generations train in hours on a single GPU. Weights, code, and the evaluation harness are openly released.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 30 of its 38 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

30 of 38 references.

References · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  5. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  6. 2025Qwen2.5-Omni Technical Reportcited by 736
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  9. 2020Contrastive Learning with Hard Negative Samplescited by 628
  10. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  11. 2019Clotho: An Audio Captioning Datasetcited by 358
  12. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
18 more
  1. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  2. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 230
  3. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 228
  4. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  5. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 116
  6. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  7. 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
  8. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  9. 2024Scaling up masked audio encoder learning for general audio classificationcited by 44
  10. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 39
  11. 2024AudioSetCaps: An Enriched Audio-Caption Dataset using Automated Generation Pipeline with Large Audio and Language Modelscited by 21
  12. 2024FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusioncited by 19
  13. 2024Cacophony: An Improved Contrastive Audio-Text Modelcited by 18
  14. 2024Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encodingcited by 14
  15. 2025M2D-CLAP: Exploring General-purpose Audio-Language Representations Beyond CLAPcited by 10
  16. 2025GLAP: General contrastive audio-text pretraining across domains and languagescited by 10
  17. 2026jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towerscited by 5
  18. 2026Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrievalcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Discriminative Axis, Not Data Volume: What a Contrastive Corpus Teaches an Audio Embeddingcited by 1
  2. 2026Tactus: Open-Vocabulary Object Recognition from Low-Cost Pressure Arrayscited by 0
  3. 2026Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservationcited by 0