# Exploring Train and Test-Time Augmentations for Audio-Language Learning

arXiv: https://arxiv.org/abs/2210.17143 · HTML: https://mlsanity.com/paper/2210.17143/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 31 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Better Aggregation in Test-Time Augmentation](https://mlsanity.com/paper/2011.11156/) (2020)
- [Improved Image Captioning via Policy Gradient optimization of SPIDEr](https://mlsanity.com/paper/1612.00370/) (2016)
- [Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation](https://mlsanity.com/paper/2002.09103/) (2020)
- [MixGen: A New Multi-Modal Data Augmentation](https://mlsanity.com/paper/2206.08358/) (2022)
- [Learning Loss for Test-Time Augmentation](https://mlsanity.com/paper/2010.11422/) (2020)
- [Audio Captioning Transformer](https://mlsanity.com/paper/2107.09817/) (2021)
- [On Metric Learning for Audio-Text Cross-Modal Retrieval](https://mlsanity.com/paper/2203.15537/) (2022)
- [SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classification](https://mlsanity.com/paper/2103.16858/) (2021)

## Built on by (most cited first, 11 of 11)

- [EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioning](https://mlsanity.com/paper/2401.17690/) (2024)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition](https://mlsanity.com/paper/2412.09501/) (2024)
- [Zero-shot audio captioning with audio-language model guidance and audio context keywords](https://mlsanity.com/paper/2311.08396/) (2023)
- [Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?](https://mlsanity.com/paper/2308.15090/) (2023)
- [EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performance](https://mlsanity.com/paper/2409.01201/) (2024)
- [Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learning](https://mlsanity.com/paper/2410.10913/) (2024)
- [Multiple Choice Learning of Low-Rank Adapters for Language Modeling](https://mlsanity.com/paper/2507.10419/) (2025)
- [Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioning](https://mlsanity.com/paper/2502.05435/) (2025)
- [Audio Difference Learning for Audio Captioning](https://mlsanity.com/paper/2309.08141/) (2023)
- [Thinking While Listening: Simple Test Time Scaling For Audio Classification](https://mlsanity.com/paper/2509.19676/) (2025)

## Abstract (arXiv)

In this paper, we aim to unveil the impact of data augmentation in audio-language multi-modal learning, which has not been explored despite its importance. We explore various augmentation methods at not only train-time but also test-time and find out that proper data augmentation can lead to substantial improvements. Specifically, applying our proposed audio-language paired augmentation PairMix, which is the first multi-modal audio-language augmentation method, outperforms the baselines for both automated audio captioning and audio-text retrieval tasks. To fully take advantage of data augmentation, we also present multi-level test-time augmentation (Multi-TTA) for the test-time. We successfully incorporate the two proposed methods and uni-modal augmentations and achieve 47.5 SPIDEr on audio captioning, which is an 18.2% relative increase over the baseline. In audio-text retrieval, the proposed methods also show an improvement in performance as well.