mlsanityData to 14 Oct 2026

Exploring Train and Test-Time Augmentations for Audio-Language Learning

arXiv 2210.17143 ↗First posted 31 Oct 2022cs.SD cs.CL eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

In this paper, we aim to unveil the impact of data augmentation in audio-language multi-modal learning, which has not been explored despite its importance. We explore various augmentation methods at not only train-time but also test-time and find out that proper data augmentation can lead to substantial improvements. Specifically, applying our proposed audio-language paired augmentation PairMix, which is the first multi-modal audio-language augmentation method, outperforms the baselines for both automated audio captioning and audio-text retrieval tasks. To fully take advantage of data augmentation, we also present multi-level test-time augmentation (Multi-TTA) for the test-time. We successfully incorporate the two proposed methods and uni-modal augmentations and achieve 47.5 SPIDEr on audio captioning, which is an 18.2% relative increase over the baseline. In audio-text retrieval, the proposed methods also show an improvement in performance as well.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 18 of its 25 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

18 of 25 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  8. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  9. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  10. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  11. 2020Better Aggregation in Test-Time Augmentationcited by 122
  12. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 113
6 more
  1. 2020Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentationcited by 61
  2. 2022MixGen: A New Multi-Modal Data Augmentationcited by 61
  3. 2020Learning Loss for Test-Time Augmentationcited by 56
  4. 2021Audio Captioning Transformercited by 55
  5. 2022On Metric Learning for Audio-Text Cross-Modal Retrievalcited by 30
  6. 2021SpecAugment++: A Hidden Space Data Augmentation Method for Acoustic Scene Classificationcited by 12

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 5; 2025: 3.

11 papers.

  1. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  2. 2024Taming Data and Transformers for Audio Generationcited by 28
  3. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognitioncited by 20
  4. 2023Zero-shot audio captioning with audio-language model guidance and audio context keywordscited by 11
  5. 2023Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval?cited by 3
  6. 2024EnCLAP++: Analyzing the EnCLAP Framework for Optimizing Automated Audio Captioning Performancecited by 3
  7. 2024Enhancing Retrieval-Augmented Audio Captioning with Generation-Assisted Multimodal Querying and Progressive Learningcited by 3
  8. 2025Multiple Choice Learning of Low-Rank Adapters for Language Modelingcited by 2
  9. 2025Unbiased Sliced Wasserstein Kernels for High-Quality Audio Captioningcited by 1
  10. 2023Audio Difference Learning for Audio Captioningcited by 0
  11. 2025Thinking While Listening: Simple Test Time Scaling For Audio Classificationcited by 0