# On Frequency-Wise Normalizations for Better Recording Device Generalization in Audio Spectrogram Transformers

arXiv: https://arxiv.org/abs/2306.11764 · HTML: https://mlsanity.com/paper/2306.11764/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 20 Jun 2023.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Two at Once: Enhancing Learning and Generalization Capacities via IBN-Net](https://mlsanity.com/paper/1807.09441/) (2018)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Batch-Instance Normalization for Adaptively Style-Invariant Neural Networks](https://mlsanity.com/paper/1805.07925/) (2018)
- [HEAR: Holistic Evaluation of Audio Representations](https://mlsanity.com/paper/2203.03022/) (2022)
- [Meta Batch-Instance Normalization for Generalizable Person Re-Identification](https://mlsanity.com/paper/2011.14670/) (2020)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representation](https://mlsanity.com/paper/2204.12260/) (2022)
- [Masked Spectrogram Prediction For Self-Supervised Audio Pre-Training](https://mlsanity.com/paper/2204.12768/) (2022)
- [Domain Generalization with Relaxed Instance Frequency-wise Normalization for Multi-device Acoustic Scene Classification](https://mlsanity.com/paper/2206.12513/) (2022)

## Abstract (arXiv)

Varying conditions between the data seen at training and at application time remain a major challenge for machine learning. We study this problem in the context of Acoustic Scene Classification (ASC) with mismatching recording devices. Previous works successfully employed frequency-wise normalization of inputs and hidden layer activations in convolutional neural networks to reduce the recording device discrepancy. The main objective of this work was to adopt frequency-wise normalization for Audio Spectrogram Transformers (ASTs), which have recently become the dominant model architecture in ASC. To this end, we first investigate how recording device characteristics are encoded in the hidden layer activations of ASTs. We find that recording device information is initially encoded in the frequency dimension; however, after the first self-attention block, it is largely transformed into the token dimension. Based on this observation, we conjecture that suppressing recording device characteristics in the input spectrogram is the most effective. We propose a frequency-centering operation for spectrograms that improves the ASC performance on unseen recording devices on average by up to 18.2 percentage points.