# Generalization in birdsong classification: impact of transfer learning methods and dataset characteristics

arXiv: https://arxiv.org/abs/2409.15383 · HTML: https://mlsanity.com/paper/2409.15383/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 21 Sep 2024.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [AST: Audio Spectrogram Transformer](https://mlsanity.com/paper/2104.01778/) (2021)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation](https://mlsanity.com/paper/2102.01243/) (2021)
- [Computational bioacoustics with deep learning: a review and roadmap](https://mlsanity.com/paper/2112.06725/) (2021)
- [Global birdsong embeddings enable superior transfer learning for bioacoustic classification](https://mlsanity.com/paper/2307.06292/) (2023)
- [Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge](https://mlsanity.com/paper/1807.05812/) (2018)
- [Transfer Learning for Speech and Language Processing](https://mlsanity.com/paper/1511.06066/) (2015)
- [Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillation](https://mlsanity.com/paper/2211.04772/) (2022)
- [Temporal Knowledge Distillation for On-device Audio Classification](https://mlsanity.com/paper/2110.14131/) (2021)
- [CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification](https://mlsanity.com/paper/2203.06760/) (2022)
- [In Search for a Generalizable Method for Source Free Domain Adaptation](https://mlsanity.com/paper/2302.06658/) (2023)
- [DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners](https://mlsanity.com/paper/2407.04082/) (2024)
- [Leveraging tropical reef, bird and unrelated sounds for superior transfer learning in marine bioacoustics](https://mlsanity.com/paper/2404.16436/) (2024)

## Built on by (most cited first, 5 of 5)

- [Clustering and novel class recognition: evaluating bioacoustic deep learning feature extractors](https://mlsanity.com/paper/2504.06710/) (2025)
- [Linear probing enables Ship-Radiated Noise recognition with pretrained audio embeddings](https://mlsanity.com/paper/2601.08358/) (2026)
- [bacpipe: a Python package to make bioacoustic deep learning models accessible](https://mlsanity.com/paper/2604.11560/) (2026)
- [Multi-Year-to-Decadal Temperature Prediction using a Machine Learning Model-Analog Framework](https://mlsanity.com/paper/2502.17583/) (2025)
- [A Bird Song Detector for improving bird identification through Deep Learning: a case study from Do~nana](https://mlsanity.com/paper/2503.15576/) (2025)

## Abstract (arXiv)

Animal sounds can be recognised automatically by machine learning, and this has an important role to play in biodiversity monitoring. Yet despite increasingly impressive capabilities, bioacoustic species classifiers still exhibit imbalanced performance across species and habitats, especially in complex soundscapes. In this study, we explore the effectiveness of transfer learning in large-scale bird sound classification across various conditions, including single- and multi-label scenarios, and across different model architectures such as CNNs and Transformers. Our experiments demonstrate that both fine-tuning and knowledge distillation yield strong performance, with cross-distillation proving particularly effective in improving in-domain performance on Xeno-canto data. However, when generalizing to soundscapes, shallow fine-tuning exhibits superior performance compared to knowledge distillation, highlighting its robustness and constrained nature. Our study further investigates how to use multi-species labels, in cases where these are present but incomplete. We advocate for more comprehensive labeling practices within the animal sound community, including annotating background species and providing temporal details, to enhance the training of robust bird sound classifiers. These findings provide insights into the optimal reuse of pretrained models for advancing automatic bioacoustic recognition.