mlsanityData to 7 Oct 2026

Optimizing Speech Multi-View Feature Fusion through Conditional Computation

arXiv 2501.08057 ↗First posted 14 Jan 2025eess.AS cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advancements have highlighted the efficacy of self-supervised learning (SSL) features in various speech-related tasks, providing lightweight and versatile multi-view speech representations. However, our study reveals that while SSL features expedite model convergence, they conflict with traditional spectral features like FBanks in terms of update directions. In response, we propose a novel generalized feature fusion framework grounded in conditional computation, featuring a gradient-sensitive gating network and a multi-stage dropout strategy. This framework mitigates feature conflicts and bolsters model robustness to multi-view input features. By integrating SSL and spectral features, our approach accelerates convergence and maintains performance on par with spectral models across multiple speech translation tasks on the MUSTC dataset.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

21 of 30 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
  9. 2024The Platonic Representation Hypothesiscited by 362
  10. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 302
  11. 2019Depth-Adaptive Transformercited by 227
  12. 2013Deep Learning of Representations: Looking Forwardcited by 84
9 more
  1. 2022Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learnerscited by 51
  2. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  3. 2016Learning Robust Features using Deep Learning for Automatic Seizure Detectioncited by 16
  4. 2023Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Predictioncited by 16
  5. 2023UnitSpeech: Speaker-adaptive Speech Synthesis with Untranscribed Datacited by 15
  6. 2022Combining Spectral and Self-Supervised Features for Low Resource Speech Recognition and Translationcited by 11
  7. 2023Exploration on HuBERT with Multiple Resolutionscited by 7
  8. 2023Modality Adaption or Regularization? A Case Study on End-to-End Speech Translationcited by 6
  9. 2023Soft Alignment of Modality Space for End-to-end Speech Translationcited by 4

Built on by

Not cited yet.