# Multi-layer attentive probing improves transfer of audio representations for bioacoustics

arXiv: https://arxiv.org/abs/2605.10494 · HTML: https://mlsanity.com/paper/2605.10494/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 11 May 2026.

## Builds on (most cited first, 19 of 19)

- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Computational bioacoustics with deep learning: a review and roadmap](https://mlsanity.com/paper/2112.06725/) (2021)
- [SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities](https://mlsanity.com/paper/2203.06849/) (2022)
- [ML-SUPERB: Multilingual Speech Universal PERformance Benchmark](https://mlsanity.com/paper/2305.10615/) (2023)
- [AVES: Animal Vocalization Encoder based on Self-Supervision](https://mlsanity.com/paper/2210.14493/) (2022)
- [EAT: Self-Supervised Pre-Training with Efficient Audio Transformer](https://mlsanity.com/paper/2401.03497/) (2024)
- [Global birdsong embeddings enable superior transfer learning for bioacoustic classification](https://mlsanity.com/paper/2307.06292/) (2023)
- [BEANS: The Benchmark of Animal Sounds](https://mlsanity.com/paper/2210.12300/) (2022)
- [BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacoustics](https://mlsanity.com/paper/2403.10380/) (2024)
- [NatureLM-audio: an Audio-Language Foundation Model for Bioacoustics](https://mlsanity.com/paper/2411.07186/) (2024)
- [ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets](https://mlsanity.com/paper/2406.08641/) (2024)
- [Can Masked Autoencoders Also Listen to Birds?](https://mlsanity.com/paper/2504.12880/) (2025)
- [Perch 2.0: The Bittern Lesson for Bioacoustics](https://mlsanity.com/paper/2508.04665/) (2025)
- [BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics](https://mlsanity.com/paper/2312.07439/) (2023)
- [Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Heads](https://mlsanity.com/paper/2308.14456/) (2023)
- [Comparing Self-Supervised Learning Models Pre-Trained on Human Speech and Animal Vocalizations for Bioacoustics Processing](https://mlsanity.com/paper/2501.05987/) (2025)
- [Robust detection of overlapping bioacoustic sound events](https://mlsanity.com/paper/2503.02389/) (2025)
- [Crossing the Species Divide: Transfer Learning from Speech to Animal Sounds](https://mlsanity.com/paper/2509.04166/) (2025)

## Built on by (most cited first, 1 of 1)

- [Beyond task performance: Decoding bioacoustic embeddings with speech features](https://mlsanity.com/paper/2606.14662/) (2026)

## Abstract (arXiv)

Probing heads map the representations learned from audio by a machine learning model to downstream task labels and are a key component in evaluating representation learning. Most bioacoustic benchmarks use a fixed, low-capacity probe, such as a linear layer on the final encoder layer. While this standardization enables model comparisons, it may bias results by overlooking the interaction between encoder features and probe design. In this work, we systematically study different probing strategies across two bioacoustic benchmarks, BEANs and BirdSet. We evaluate last- and multi-layer probing, across linear and attention probes. We show that larger probe heads that leverage time information have superior performance. Our results suggest that current benchmarks may misrepresent encoder quality when relying on a last-layer probing setup. Multi-layer probing improves downstream task performance across all tested models, while attention probing has superior performance to linear probing for transformer models.