# Play It Back: Iterative Attention for Audio Recognition

arXiv: https://arxiv.org/abs/2210.11328 · HTML: https://mlsanity.com/paper/2210.11328/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 37 references that are in the map.
- First posted on arXiv 20 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation](https://mlsanity.com/paper/2102.01243/) (2021)
- [MAE-AST: Masked Autoencoding Audio Spectrogram Transformer](https://mlsanity.com/paper/2203.16691/) (2022)
- [PolyViT: Co-training Vision Transformers on Images, Videos and Audio](https://mlsanity.com/paper/2111.12993/) (2021)
- [Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks](https://mlsanity.com/paper/1809.03355/) (2018)
- [Slow-Fast Auditory Streams For Audio Recognition](https://mlsanity.com/paper/2103.03516/) (2021)
- [Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training](https://mlsanity.com/paper/2204.12768/) (2022)
- [A Comparison of Five Multiple Instance Learning Pooling Functions for Sound Event Detection with Weak Labeling](https://mlsanity.com/paper/1810.09050/) (2018)
- [Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimization](https://mlsanity.com/paper/1912.04761/) (2019)
- [Visual Concept Recognition and Localization via Iterative Introspection](https://mlsanity.com/paper/1603.04186/) (2016)
- [AutoScaler: Scale-Attention Networks for Visual Correspondence](https://mlsanity.com/paper/1611.05837/) (2016)
- [Learning Temporal Resolution in Spectrogram for Audio Classification](https://mlsanity.com/paper/2210.01719/) (2022)

## Built on by (most cited first, 3 of 3)

- [Audiovisual Masked Autoencoders](https://mlsanity.com/paper/2212.05922/) (2022)
- [Multimodal Distillation for Egocentric Action Recognition](https://mlsanity.com/paper/2307.07483/) (2023)
- [Zorro: the masked multimodal transformer](https://mlsanity.com/paper/2301.09595/) (2023)

## Abstract (arXiv)

A key function of auditory cognition is the association of characteristic sounds with their corresponding semantics over time. Humans attempting to discriminate between fine-grained audio categories, often replay the same discriminative sounds to increase their prediction confidence. We propose an end-to-end attention-based architecture that through selective repetition attends over the most discriminative sounds across the audio sequence. Our model initially uses the full audio sequence and iteratively refines the temporal segments replayed based on slot attention. At each playback, the selected segments are replayed using a smaller hop length which represents higher resolution features within these segments. We show that our method can consistently achieve state-of-the-art performance across three audio-classification benchmarks: AudioSet, VGG-Sound, and EPIC-KITCHENS-100.