mlsanityData to 14 Oct 2026

What Makes a Good Layer? Assessing the Layer-Wise Intrinsic Properties of Music Foundation Models

arXiv 2608.14819 ↗First posted 14 Aug 2026cs.SD cs.LG eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

Music foundation models are commonly used as frozen audio feature extractors, yet selecting which layer to extract from remains largely heuristic. Current practice defaults to fixed depths or multi-layer fusion, with limited understanding of why certain layers transfer better across downstream tasks or how representation quality varies with depth and pre-training paradigm. We conduct a systematic layer-wise analysis of 12 music foundation models spanning three pre-training paradigms (masked modeling, autoregressive modeling, and contrastive learning), characterizing their hidden representations through intrinsic geometric and transformation-based properties. Correlating label-free representation-quality metrics with layer-wise performance across 15 downstream tasks, we find that several metrics track layer quality for genre classification, emotion recognition, automatic tagging, and beat tracking, albeit with varying strength across tasks and pre-training paradigms. However, all metrics fail on tonal tasks such as key estimation and chord recognition, indicating that no single property serves as a general proxy for representation quality across music information retrieval tasks. To address this gap, we introduce a pitch-transposition equivariance measure that captures properties missed by these standard metrics, providing a consistent indicator of tonal quality across model families. Finally, we show that intrinsic metrics can serve as effective proxies for layer selection, matching or outperforming trainable multi-layer fusion methods, particularly in limited-data settings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 59 of its 86 references that are in the map. First posted on arXiv 14 Aug 2026.

Builds on

59 of 86 references.

References · 59

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  3. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  4. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  5. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2019Designing and Interpreting Probes with Control Taskscited by 640
  8. 2020Jukebox: A Generative Model for Musiccited by 542
  9. 2023Simple and Controllable Music Generationcited by 488
  10. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  11. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  12. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
38 more
  1. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 250
  2. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  3. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  4. 2020Information-Theoretic Probing for Linguistic Structurecited by 207
  5. 2023MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Trainingcited by 181
  6. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  7. 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 153
  8. 2023The geometry of hidden representations of large transformer modelscited by 112
  9. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 87
  10. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  11. 2025YuE: Scaling Open Foundation Models for Long-Form Music Generationcited by 71
  12. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 68
  13. 2021Codified audio language modeling learns useful representations for music information retrievalcited by 64
  14. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 63
  15. 2024Emergence of a High-Dimensional Abstraction Phase in Language Transformerscited by 56
  16. 2023A Foundation Model for Music Informaticscited by 46
  17. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  18. 2023The Shape of Learning: Anisotropy and Intrinsic Dimensions in Transformer-Based Modelscited by 39
  19. 2023MARBLE: Music Audio Representation Benchmark for Universal Evaluationcited by 38
  20. 2026Temporal Straightening for Latent Planningcited by 38
  21. 2024Foundation Models for Music: A Surveycited by 37
  22. 2023Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural languagecited by 32
  23. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  24. 2023LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architecturescited by 27
  25. 2015Deep Learning and Music Adversariescited by 20
  26. 2020Disentangled Multidimensional Metric Learning for Music Similaritycited by 16
  27. 2018Persistent Homology and the Upper Box Dimensioncited by 15
  28. 2024Do Music Generation Models Encode Music Theory?cited by 13
  29. 2025Discovering and Steering Interpretable Concepts in Large Generative Music Modelscited by 13
  30. 2023Unsupervised Embedding Quality Evaluationcited by 9
  31. 2024Learning predictable and robust neural representations by straightening image sequencescited by 9
  32. 2024STONE: Self-supervised Tonality Estimatorcited by 8
  33. 2026Context Structure Reshapes the Representational Geometry of Language Modelscited by 8
  34. 2025Towards a Unified Representation Evaluation Framework Beyond Downstream Taskscited by 7
  35. 2024Music Foundation Model as Generic Booster for Music Downstream Taskscited by 6
  36. 2024Interface Design for Self-Supervised Speech Modelscited by 5
  37. 2024Towards Automatic Assessment of Self-Supervised Speech Models using Rankcited by 5
  38. 2025Towards Early Prediction of Self-Supervised Speech Model Performancecited by 5

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.