# Lightweight and Generalizable Acoustic Scene Representations via Contrastive Fine-Tuning and Distillation

arXiv: https://arxiv.org/abs/2510.03728 · HTML: https://mlsanity.com/paper/2510.03728/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Understanding and Improving Layer Normalization](https://mlsanity.com/paper/1911.07013/) (2019)
- [Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks](https://mlsanity.com/paper/1702.05870/) (2017)
- [Domain Generalization with Relaxed Instance Frequency-wise Normalization for Multi-device Acoustic Scene Classification](https://mlsanity.com/paper/2206.12513/) (2022)
- [Device-Robust Acoustic Scene Classification via Impulse Response Augmentation](https://mlsanity.com/paper/2305.07499/) (2023)
- [Data-Efficient Low-Complexity Acoustic Scene Classification in the DCASE 2024 Challenge](https://mlsanity.com/paper/2405.10018/) (2024)
- [Data-Efficient Low-Complexity Acoustic Scene Classification via Distilling and Progressive Pruning](https://mlsanity.com/paper/2410.20775/) (2024)
- [Low-Complexity Acoustic Scene Classification with Device Information in the DCASE 2025 Challenge](https://mlsanity.com/paper/2505.01747/) (2025)
- [Creating a Good Teacher for Knowledge Distillation in Acoustic Scene Classification](https://mlsanity.com/paper/2503.11363/) (2025)
- [SonicSieve: Bringing Directional Speech Extraction to Smartphones Using Acoustic Microstructures](https://mlsanity.com/paper/2504.10793/) (2025)

## Abstract (arXiv)

Acoustic scene classification (ASC) models on edge devices typically operate under fixed class assumptions, lacking the transferability needed for real-world applications that require adaptation to new or refined acoustic categories. We propose ContrastASC, which learns generalizable acoustic scene representations by structuring the embedding space to preserve semantic relationships between scenes, enabling adaptation to unseen categories without retraining. Our approach combines supervised contrastive fine-tuning of pre-trained models with contrastive representation distillation to transfer this structured knowledge to compact student models. Our evaluation shows that ContrastASC demonstrates improved few-shot adaptation to unseen categories while maintaining strong closed-set performance.