# Fairness and underspecification in acoustic scene classification: The case for disaggregated evaluations

arXiv: https://arxiv.org/abs/2110.01506 · HTML: https://mlsanity.com/paper/2110.01506/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 30 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Builds on (most cited first, 13 of 13)

- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [A Survey on the Explainability of Supervised Machine Learning](https://mlsanity.com/paper/2011.07876/) (2020)
- [A multi-device dataset for urban acoustic scene classification](https://mlsanity.com/paper/1807.09840/) (2018)
- [Acoustic scene classification in DCASE 2020 Challenge: generalization across devices and low complexity solutions](https://mlsanity.com/paper/2005.14623/) (2020)
- [Acoustic Scene Classification](https://mlsanity.com/paper/1411.3715/) (2014)
- [Neural Anisotropy Directions](https://mlsanity.com/paper/2006.09717/) (2020)
- [Integrating the Data Augmentation Scheme with Various Classifiers for Acoustic Scene Modeling](https://mlsanity.com/paper/1907.06639/) (2019)
- [Unsupervised adversarial domain adaptation for acoustic scene classification](https://mlsanity.com/paper/1808.05777/) (2018)
- [Unsupervised Adversarial Domain Adaptation Based On The Wasserstein Distance For Acoustic Scene Classification](https://mlsanity.com/paper/1904.10678/) (2019)
- [CAA-Net: Conditional Atrous CNNs with Attention for Explainable Device-robust Acoustic Scene Classification](https://mlsanity.com/paper/2011.09299/) (2020)
- [A neural anisotropic view of underspecification in deep learning](https://mlsanity.com/paper/2104.14372/) (2021)

## Built on by (most cited first, 4 of 4)

- [Bringing the Discussion of Minima Sharpness to the Audio Domain: a Filter-Normalised Evaluation for Acoustic Scene Classification](https://mlsanity.com/paper/2309.16369/) (2023)
- [Towards Friendly AI: A Comprehensive Review and New Perspectives on Human-AI Alignment](https://mlsanity.com/paper/2412.15114/) (2024)
- [Climate Change & Computer Audition: A Call to Action and Overview on Audio Intelligence to Help Save the Planet](https://mlsanity.com/paper/2203.06064/) (2022)
- [Fatigue Prediction in Outdoor Running Conditions using Audio Data](https://mlsanity.com/paper/2205.04343/) (2022)

## Abstract (arXiv)

Underspecification and fairness in machine learning (ML) applications have recently become two prominent issues in the ML community. Acoustic scene classification (ASC) applications have so far remained unaffected by this discussion, but are now becoming increasingly used in real-world systems where fairness and reliability are critical aspects. In this work, we argue for the need of a more holistic evaluation process for ASC models through disaggregated evaluations. This entails taking into account performance differences across several factors, such as city, location, and recording device. Although these factors play a well-understood role in the performance of ASC models, most works report single evaluation metrics taking into account all different strata of a particular dataset. We argue that metrics computed on specific sub-populations of the underlying data contain valuable information about the expected real-world behaviour of proposed systems, and their reporting could improve the transparency and trustability of such systems. We demonstrate the effectiveness of the proposed evaluation process in uncovering underspecification and fairness problems exhibited by several standard ML architectures when trained on two widely-used ASC datasets. Our evaluation shows that all examined architectures exhibit large biases across all factors taken into consideration, and in particular with respect to the recording location. Additionally, different architectures exhibit different biases even though they are trained with the same experimental configurations.