Anchoring Speech with Semantics: A Multimodal Adapter Mechanism for Automatic Speech Recognition in Low-Resource Languages
Not on the map. Why?
Abstract (arXiv)
Low-resource ASR remains difficult because scarce transcripts provide limited supervised evidence for target-side generation. To address this gap, we propose SAMA-ASR, a lightweight adapter mechanism that augments the decoder with semantic anchors from auxiliary translations and an acoustic anchor from speech; in principle, the mechanism can be applied to similar encoder--decoder multitask speech models. Through cross-modal adaptation, SAMA-ASR conditions decoder states on translation-derived semantic embeddings and a speech embedding, combining utterance-level meaning with speech-grounded evidence before token prediction. At evaluation time, these semantic anchors can be generated automatically by an upstream speech-to-text translator rather than supplied as oracle translations. Experiments on two 30-hour datasets covering the low-resource Sinitic varieties Taiwanese Hokkien and Hakka show that SAMA-ASR improves over acoustic, prior prompt-based, and semantic-only translation-guided baselines and remains effective in practical automatic semantic-anchor settings; translator-capacity analyses show that useful semantic anchors can be produced by a compact ST model.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 29 Aug 2026.
Builds on
24 of 41 references.
References · 24
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
- 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
- 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
- 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
- 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
- 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
- 2020Dual-decoder Transformer for Joint Automatic Speech Recognition and Multilingual Speech Translationcited by 54
- 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
- 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 35
- 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 35
12 more
- 2023Can Contextual Biasing Remain Effective with Whisper and GPT-2?cited by 21
- 2018Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognitioncited by 18
- 2023STT4SG-350: A Speech Corpus for All Swiss German Dialect Regionscited by 18
- 2022Speech Resources in the Tamasheq Languagecited by 13
- 2024Enhancing Low-Resource ASR through Versatile TTS: Bridging the Data Gapcited by 11
- 2022Distilling a Pretrained Language Model to a Multilingual ASR Modelcited by 7
- 2024Enabling ASR for Low-Resource Languages: A Comprehensive Dataset Creation Approachcited by 7
- 2024Let's Fuse Step by Step: A Generative Fusion Decoding Algorithm with LLMs for Robust and Instruction-Aware ASR and OCRcited by 4
- 2023HK-LegiCoST: Leveraging Non-Verbatim Transcripts for Speech Translationcited by 3
- 2024Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recogniserscited by 3
- 2026TG-ASR: Translation-Guided Learning with Parallel Gated Cross Attention for Low-Resource Automatic Speech Recognitioncited by 2
- 2026Efficient Dialect-Aware Modeling and Conditioning for Low-Resource Taiwanese Hakka Speech Processingcited by 1
Built on by
Not cited yet.