# Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer

arXiv: https://arxiv.org/abs/2112.08995 · HTML: https://mlsanity.com/paper/2112.08995/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 60 references that are in the map.
- First posted on arXiv 16 Dec 2021.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [Large Batch Training of Convolutional Networks](https://mlsanity.com/paper/1708.03888/) (2017)
- [Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling](https://mlsanity.com/paper/2102.06183/) (2021)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [Self-Supervised MultiModal Versatile Networks](https://mlsanity.com/paper/2006.16228/) (2020)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [Very Deep Convolutional Neural Networks for Raw Waveforms](https://mlsanity.com/paper/1610.00087/) (2016)
- [Audio Retrieval with Natural Language Queries](https://mlsanity.com/paper/2105.02192/) (2021)
- [Multimodal Self-Supervised Learning of General Audio Representations](https://mlsanity.com/paper/2104.12807/) (2021)
- [ESResNet: Environmental Sound Classification Based on Visual Domain Models](https://mlsanity.com/paper/2004.07301/) (2020)
- [Adaptive pooling operators for weakly labeled sound event detection](https://mlsanity.com/paper/1804.10070/) (2018)
- [Knowledge Transfer from Weakly Labeled Audio using Convolutional Neural Network for Sound Events and Scenes](https://mlsanity.com/paper/1711.01369/) (2017)
- [Multimodal Pivots for Image Caption Translation](https://mlsanity.com/paper/1601.03916/) (2016)
- [Audio Set classification with attention model: A probabilistic perspective](https://mlsanity.com/paper/1711.00927/) (2017)
- [Zero-resource Machine Translation by Multimodal Encoder-decoder Network with Multimedia Pivot](https://mlsanity.com/paper/1611.04503/) (2016)
- [Audio Event and Scene Recognition: A Unified Approach using Strongly and Weakly Labeled Data](https://mlsanity.com/paper/1611.04871/) (2016)

## Built on by (most cited first, 17 of 17)

- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [Gramian Multimodal Representation Learning and Alignment](https://mlsanity.com/paper/2412.11959/) (2024)
- [ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound](https://mlsanity.com/paper/2204.02874/) (2022)
- [Contrastive Audio-Language Learning for Music](https://mlsanity.com/paper/2208.12208/) (2022)
- [Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learning](https://mlsanity.com/paper/2309.11500/) (2023)
- [Multimodal Knowledge Alignment with Reinforcement Learning](https://mlsanity.com/paper/2205.12630/) (2022)
- [Principled Multimodal Representation Learning](https://mlsanity.com/paper/2507.17343/) (2025)
- [A TRIANGLE Enables Multimodal Alignment Beyond Cosine Similarity](https://mlsanity.com/paper/2509.24734/) (2025)
- [Harvesting Event Schemas from Large Language Models](https://mlsanity.com/paper/2305.07280/) (2023)
- [Unsupervised Improvement of Audio-Text Cross-Modal Representations](https://mlsanity.com/paper/2305.01864/) (2023)
- [Anaphor Assisted Document-Level Relation Extraction](https://mlsanity.com/paper/2310.18604/) (2023)
- [Translating speech with just images](https://mlsanity.com/paper/2406.07133/) (2024)
- [Enhancing Audio-Language Models through Self-Supervised Post-Training with Text-Audio Pairs](https://mlsanity.com/paper/2408.09269/) (2024)
- [v-HUB: A Benchmark for Video Humor Understanding from Vision and Sound](https://mlsanity.com/paper/2509.25773/) (2025)
- [Acoustic Identification of Ae. aegypti Mosquitoes using Smartphone Apps and Residual Convolutional Neural Networks](https://mlsanity.com/paper/2306.10091/) (2023)
- [Hypergraph-Regularized Gramian Volumes for Multimodal Retrieval](https://mlsanity.com/paper/2609.15320/) (2026)

## Abstract (arXiv)

Machines that can represent and describe environmental soundscapes have practical potential, e.g., for audio tagging and captioning systems. Prevailing learning paradigms have been relying on parallel audio-text data, which is, however, scarcely available on the web. We propose VIP-ANT that induces Audio-Text alignment without using any parallel audio-text data. Our key idea is to share the image modality between bi-modal image-text representations and bi-modal image-audio representations; the image modality functions as a pivot and connects audio and text in a tri-modal embedding space implicitly. In a difficult zero-shot setting with no paired audio-text data, our model demonstrates state-of-the-art zero-shot performance on the ESC50 and US8K audio classification tasks, and even surpasses the supervised state of the art for Clotho caption retrieval (with audio queries) by 2.2% R@1. We further investigate cases of minimal audio-text supervision, finding that, e.g., just a few hundred supervised audio-text pairs increase the zero-shot audio classification accuracy by 8% on US8K. However, to match human parity on some zero-shot tasks, our empirical scaling experiments suggest that we would need about 221 ≈ 2M supervised audio-caption pairs. Our work opens up new avenues for learning audio-text connections with little to no parallel audio-text data.