# Massive Sound Embedding Benchmark (MSEB)

arXiv: https://arxiv.org/abs/2602.07143 · HTML: https://mlsanity.com/paper/2602.07143/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 6 Feb 2026.

## Builds on (most cited first, 23 of 23)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages](https://mlsanity.com/paper/2204.08582/) (2022)
- [Gemini Embedding: Generalizable Embeddings from Gemini](https://mlsanity.com/paper/2503.07891/) (2025)
- [Towards Learning a Universal Non-Semantic Representation of Speech](https://mlsanity.com/paper/2002.12764/) (2020)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [Global birdsong embeddings enable superior transfer learning for bioacoustic classification](https://mlsanity.com/paper/2307.06292/) (2023)
- [End-to-End Text-Dependent Speaker Verification](https://mlsanity.com/paper/1509.08062/) (2015)
- [Unsupervised Learning of Semantic Audio Representations](https://mlsanity.com/paper/1711.02209/) (2017)
- [BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics](https://mlsanity.com/paper/2403.10380/) (2024)
- [XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages](https://mlsanity.com/paper/2305.11938/) (2023)
- [KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation](https://mlsanity.com/paper/2502.15602/) (2025)
- [Speech-MASSIVE: A Multilingual Speech Dataset for SLU and Beyond](https://mlsanity.com/paper/2408.03900/) (2024)
- [MIEB: Massive Image Embedding Benchmark](https://mlsanity.com/paper/2504.10471/) (2025)

## Built on by (most cited first, 9 of 9)

- [Multi-Vector Index Compression in Any Modality](https://mlsanity.com/paper/2602.21202/) (2026)
- [Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini](https://mlsanity.com/paper/2605.27295/) (2026)
- [GlobeAudio: A Multilingual Multicultural Benchmark for Naturalistic Evaluation of Large Audio-Language Models](https://mlsanity.com/paper/2606.08194/) (2026)
- [SQuTR: A Robustness Benchmark for Spoken Query to Text Retrieval under Acoustic Noise](https://mlsanity.com/paper/2602.12783/) (2026)
- [ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval](https://mlsanity.com/paper/2605.03361/) (2026)
- [ATIR: Towards Audio-Text Interleaved Contextual Retrieval](https://mlsanity.com/paper/2604.20267/) (2026)
- [Benchmarking LLMs on the Massive Sound Embedding Benchmark (MSEB)](https://mlsanity.com/paper/2605.04556/) (2026)
- [INSPIRE: A Benchmark for Instruction-Aware Speech Retrieval](https://mlsanity.com/paper/2608.16203/) (2026)
- [LAYERSCOPE: A Layerwise Characterization of Video and Multimodal Learned Representations](https://mlsanity.com/paper/2609.28086/) (2026)

## Abstract (arXiv)

Audio is a critical component of multimodal perception, and any truly intelligent system must demonstrate a wide range of auditory capabilities. These capabilities include transcription, classification, retrieval, reasoning, segmentation, clustering, reranking, and reconstruction. Fundamentally, each task involves transforming a raw audio signal into a meaningful 'embedding' - be it a single vector, a sequence of continuous or discrete representations, or another structured form - which then serves as the basis for generating the task's final response. To accelerate progress towards robust machine auditory intelligence, we present the Massive Sound Embedding Benchmark (MSEB): an extensible framework designed to evaluate the auditory components of any multimodal system. In its first release, MSEB offers a comprehensive suite of eight core tasks, with more planned for the future, supported by diverse datasets, including the new, large-scale Simple Voice Questions (SVQ) dataset. Our initial experiments establish clear performance headrooms, highlighting the significant opportunity to improve real-world multimodal experiences where audio is a core signal. We encourage the research community to use MSEB to assess their algorithms and contribute to its growth. The library is publicly hosted at github.