# Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

arXiv: https://arxiv.org/abs/2607.18666 · HTML: https://mlsanity.com/paper/2607.18666/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 38 references that are in the map.
- First posted on arXiv 21 Jul 2026.

## Builds on (most cited first, 30 of 30)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Contrastive Learning with Hard Negative Samples](https://mlsanity.com/paper/2010.04592/) (2020)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Ranking](https://mlsanity.com/paper/2601.04720/) (2026)
- [VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks](https://mlsanity.com/paper/2410.05160/) (2024)
- [ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities](https://mlsanity.com/paper/2305.11172/) (2023)
- [VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents](https://mlsanity.com/paper/2507.04590/) (2025)
- [LLaMA Pro: Progressive LLaMA with Block Expansion](https://mlsanity.com/paper/2401.02415/) (2024)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers](https://mlsanity.com/paper/2307.03183/) (2023)
- [Scaling up masked audio encoder learning for general audio classification](https://mlsanity.com/paper/2406.06992/) (2024)
- [OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces](https://mlsanity.com/paper/2407.11895/) (2024)
- [AudioSetCaps: An Enriched Audio-Caption Dataset using Automated Generation Pipeline with Large Audio and Language Models](https://mlsanity.com/paper/2411.18953/) (2024)
- [FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion](https://mlsanity.com/paper/2405.04883/) (2024)
- [Cacophony: An Improved Contrastive Audio-Text Model](https://mlsanity.com/paper/2402.06986/) (2024)
- [Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encoding](https://mlsanity.com/paper/2406.13275/) (2024)
- [M2D-CLAP: Exploring General-purpose Audio-Language Representations Beyond CLAP](https://mlsanity.com/paper/2503.22104/) (2025)
- [GLAP: General contrastive audio-text pretraining across domains and languages](https://mlsanity.com/paper/2506.11350/) (2025)
- [jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers](https://mlsanity.com/paper/2605.08384/) (2026)
- [Omni-Embed-Audio: Leveraging Multimodal LLMs for Robust Audio-Text Retrieval](https://mlsanity.com/paper/2604.18360/) (2026)

## Built on by (most cited first, 3 of 3)

- [Discriminative Axis, Not Data Volume: What a Contrastive Corpus Teaches an Audio Embedding](https://mlsanity.com/paper/2608.01560/) (2026)
- [Tactus: Open-Vocabulary Object Recognition from Low-Cost Pressure Arrays](https://mlsanity.com/paper/2608.04043/) (2026)
- [Modality-Gated Deep Adapters: Adding a Modality to a Frozen Embedding Model with Exact Preservation](https://mlsanity.com/paper/2609.26182/) (2026)

## Abstract (arXiv)

A single embedding space that covers text, images, video, and audio lets one index serve every query a user can pose. Embedding models built on vision-language backbones now lead text/image/video retrieval benchmarks but lack audio entirely, while audio-text retrieval is led by specialist systems that serve no other modality. We present the Fusion Embedding family, which adds audio to a frozen vision-language embedding base whose parameters are never updated: generation 1 (fusion-embedding-1) trains only a 16.4M-parameter connector between a frozen audio tower and the frozen base, and generation 2 (fusion-embedding-2) adds modality-gated deep adapters (44.2M parameters) whose branch never executes on text, image, or video inputs: their outputs are bit-for-bit those of the released base, verified after every training run. Because the base already binds text, images, and video, aligning audio to text alone makes audio-image retrieval emerge, with zero paired audio-visual training data. Alongside the recipe we map its design space with controlled negative results (rewriting training captions with an LLM, substituting a leaderboard-stronger audio tower, and widening the connector each reduce retrieval) and with training-protocol findings that we expect to transfer to any frozen decoder-LM embedding backbone. Both generations train in hours on a single GPU. Weights, code, and the evaluation harness are openly released.