# Coincidence, Categorization, and Consolidation: Learning to Recognize Sounds with Minimal Supervision

arXiv: https://arxiv.org/abs/1911.05894 · HTML: https://mlsanity.com/paper/1911.05894/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2019.

## Built on by (most cited first, 27 of 27)

- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Infinity: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis](https://mlsanity.com/paper/2412.04431/) (2024)
- [Image and Video Tokenization with Binary Spherical Quantization](https://mlsanity.com/paper/2406.07548/) (2024)
- [Broaden Your Views for Self-Supervised Video Learning](https://mlsanity.com/paper/2103.16559/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Multi-Format Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2103.06508/) (2021)
- [Universal Paralinguistic Speech Representations Using Self-Supervised Conformers](https://mlsanity.com/paper/2110.04621/) (2021)
- [AudioScopeV2: Audio-Visual Attention Architectures for Calibrated Open-Domain On-Screen Sound Separation](https://mlsanity.com/paper/2207.10141/) (2022)
- [Conformer-Based Self-Supervised Learning for Non-Speech Audio Tasks](https://mlsanity.com/paper/2110.07313/) (2021)
- [Egocentric Auditory Attention Localization in Conversations](https://mlsanity.com/paper/2303.16024/) (2023)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [WeTok: Powerful Discrete Tokenization for High-Fidelity Visual Reconstruction](https://mlsanity.com/paper/2508.05599/) (2025)
- [Self-Supervised Learning from Automatically Separated Sound Scenes](https://mlsanity.com/paper/2105.02132/) (2021)
- [Spherical Leech Quantization for Visual Tokenization and Generation](https://mlsanity.com/paper/2512.14697/) (2025)
- [Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention](https://mlsanity.com/paper/2106.09669/) (2021)
- [BitDance: Scaling Autoregressive Generative Models with Binary Tokens](https://mlsanity.com/paper/2602.14041/) (2026)
- [Human Activity Recognition Using Self-Supervised Representations of Wearable Data](https://mlsanity.com/paper/2304.14912/) (2023)
- [A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2412.00049/) (2024)
- [Semantic One-Dimensional Tokenizer for Image Reconstruction and Generation](https://mlsanity.com/paper/2603.16373/) (2026)
- [Semi-Supervised Audio Representation Learning for Modeling Beehive Strengths](https://mlsanity.com/paper/2105.10536/) (2021)
- [Human Activity Recognition on wrist-worn accelerometers using self-supervised neural networks](https://mlsanity.com/paper/2112.12272/) (2021)
- [A Study on Robustness to Perturbations for Representations of Environmental Sound](https://mlsanity.com/paper/2203.10425/) (2022)
- [Identifying Spatio-Temporal Drivers of Extreme Events](https://mlsanity.com/paper/2410.24075/) (2024)
- [Expedition: A System for the Unsupervised Learning of a Hierarchy of Concepts](https://mlsanity.com/paper/2112.09348/) (2021)
- [Recent Advances and Challenges in Deep Audio-Visual Correlation Learning](https://mlsanity.com/paper/2202.13673/) (2022)
- [UniWeTok: An Unified Binary Tokenizer with Codebook Size mathit2128 for Unified Multimodal Large Language Model](https://mlsanity.com/paper/2602.14178/) (2026)

## Abstract (arXiv)

Humans do not acquire perceptual abilities in the way we train machines. While machine learning algorithms typically operate on large collections of randomly-chosen, explicitly-labeled examples, human acquisition relies more heavily on multimodal unsupervised learning (as infants) and active learning (as children). With this motivation, we present a learning framework for sound representation and recognition that combines (i) a self-supervised objective based on a general notion of unimodal and cross-modal coincidence, (ii) a clustering objective that reflects our need to impose categorical structure on our experiences, and (iii) a cluster-based active learning procedure that solicits targeted weak supervision to consolidate categories into relevant semantic classes. By training a combined sound embedding/clustering/classification network according to these criteria, we achieve a new state-of-the-art unsupervised audio representation and demonstrate up to a 20-fold reduction in the number of labels required to reach a desired classification performance.