# A Framework for Generative and Contrastive Learning of Audio Representations

arXiv: https://arxiv.org/abs/2010.11459 · HTML: https://mlsanity.com/paper/2010.11459/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 15 of 15)

- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [Audio Transformers](https://mlsanity.com/paper/2105.00335/) (2021)
- [A Generative Model for Raw Audio Using Transformer Architectures](https://mlsanity.com/paper/2106.16036/) (2021)
- [Towards Signal Processing In Large Language Models](https://mlsanity.com/paper/2406.10254/) (2024)
- [Whisper-GPT -- Continuous Discrete Hybrid Representation Language Models For Speech And Music](https://mlsanity.com/paper/2412.11449/) (2024)
- [Attention is All You Need? Good Embeddings with Statistics are enough:Large Scale Audio Understanding without Transformers/ Convolutions/ BERTs/ Mixers/ Attention/ RNNs or ....](https://mlsanity.com/paper/2110.03183/) (2021)
- [Conformer LLMs -- Convolution Augmented Large Language Models](https://mlsanity.com/paper/2307.00461/) (2023)
- [Adaptive Large Language Models By Layerwise Attention Shortcuts](https://mlsanity.com/paper/2409.10870/) (2024)
- [Wavelet GPT: Wavelet Inspired Large Language Models](https://mlsanity.com/paper/2409.12924/) (2024)
- [Large Language Models Implicitly Learn to See and Hear Just By Reading](https://mlsanity.com/paper/2505.17091/) (2025)
- [Enhancing Audio Perception of Music By AI Picked Room Acoustics](https://mlsanity.com/paper/2208.07994/) (2022)
- [Content Adaptive Front End For Audio Classification](https://mlsanity.com/paper/2303.10446/) (2023)
- [Neural Architectures Learning Fourier Transforms, Signal Processing and Much More....](https://mlsanity.com/paper/2308.10388/) (2023)
- [Diverse Audio Embeddings -- Bringing Features Back Outperforms CLAP!](https://mlsanity.com/paper/2309.08751/) (2023)
- [Thinking While Listening: Simple Test Time Scaling For Audio Classification](https://mlsanity.com/paper/2509.19676/) (2025)

## Abstract (arXiv)

In this paper, we present a framework for contrastive learning for audio representations, in a self supervised frame work without access to any ground truth labels. The core idea in self supervised contrastive learning is to map an audio signal and its various augmented versions (representative of salient aspects of audio like pitch, timbre etc.) to a space where they are close together, and are separated from other different signals. In addition we also explore generative models based on state of the art transformer based architectures for learning latent spaces for audio signals, without access to any labels. Here, we map audio signals on a smaller scale to discrete dictionary elements and train transformers to predict the next dictionary element. We only use data as a method of supervision, bypassing the need of labels needed to act as a supervision for training the deep neural networks. We then use a linear classifier head in order to evaluate the performance of our models, for both self supervised contrastive and generative transformer based representations that are learned. Our system achieves considerable performance, compared to a fully supervised method, with access to ground truth labels to train the neural network model. These representations, with avail-ability of large scale audio data show promise in various tasks for audio understanding tasks