mlsanityData to 19 Oct 2026

Rethinking Speech Foundation Model Fine-tuning: Better SFT or Better Match?

arXiv 2607.13864 ↗First posted 15 Jul 2026cs.SDcited by 0

On the map: Speech and audio language models → Speech recognition models and training

Abstract (arXiv)

Supervised fine-tuning (SFT) is widely used to adapt self-supervised speech representations to downstream classification tasks. Small gains observed under a single pretrained checkpoint are often interpreted as method-level improvements, i.e., a higher attainable performance ceiling. We show that such conclusions are not always reliable because SFT outcomes depend strongly on the specific pretrained instance. We conduct a systematic study on 3 SUPERB classification tasks, evaluating 8 SFT variants across 9 pretrained checkpoints from wav2vec~2.0, HuBERT, and WavLM, with multi-seed repetitions on representative base-scale models. We find that the identity of the statistically indistinguishable top-group SFT recipe is often checkpoint-dependent, with limited transferability across pretrained instances. These findings suggest that many reported downstream gains reflect instance and seed dependent elicitation match, rather than universally improving the attainable performance ceiling.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Speech recognition models and training, one of 11 directions in Speech and audio language models. Builds on 16 of its 21 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

16 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing26 Oct
  2. 2021Leveraging Acoustic and Linguistic Embeddings from Pretrained speech and language Models for Intent Classification15 Feb

Other references · 14

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 4,385
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,762
  3. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 793
  4. 2020What Happens To BERT Embeddings During Fine-tuning?cited by 115
  5. 2025On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Modelscited by 74
  6. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 61
  7. 2020Improving End-to-End Speech-to-Intent Classification with Reptilecited by 18
  8. 2023Exploring Binary Classification Loss For Speaker Verificationcited by 16
  9. 2024Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decodingcited by 10
  10. 2024Efficient Adapter Tuning of Pre-trained Speech Models for Automatic Speaker Verificationcited by 8
  11. 2023A Comprehensive Study on the Effectiveness of ASR Representations for Noise-Robust Speech Emotion Recognitioncited by 5
  12. 2024MOS-FAD: Improving Fake Audio Detection Via Automatic Mean Opinion Score Predictioncited by 3
2 more
  1. 2025Bridging Speech Emotion Recognition and Personality: Dataset and Temporal Interaction Condition Networkcited by 1
  2. 2025SONAR: Self-Distilled Continual Pre-training for Domain Adaptive Audio Representationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  2. 2024Rethinking Data Selection at Scale: Random Selection is Almost All You Need
  3. 2021Layer-wise Analysis of a Self-supervised Speech Representation Model
  4. 2024Less Forgetting for Better Generalization: Exploring Continual-learning Fine-tuning Methods for Speech Self-supervised Representations
  5. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  6. 2026What Should SFT Learn? Conditional Information Gain for Generalizable Post-Training
  7. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study
  8. 2026A Mechanistic Investigation of Supervised Fine Tuning