SparQLe: Speech Queries to Text Translation Through LLMs
Not on the map. Why?
Abstract (arXiv)
With the growing influence of Large Language Models (LLMs), there is increasing interest in integrating speech representations with them to enable more seamless multi-modal processing and speech understanding. This study introduces a novel approach that combines self-supervised speech representations with instruction-tuned LLMs for speech-to-text translation. The proposed approach leverages a modality adapter to align extracted speech features with instruction-tuned LLMs using English speech data. Our experiments demonstrate that this method effectively preserves the semantic content of the input speech and serves as an effective bridge between self-supervised speech models and instruction-tuned LLMs, offering a promising approach for various speech understanding applications.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 28 of its 38 references that are in the map. First posted on arXiv 13 Feb 2025.
Builds on
28 of 38 references.
References · 28
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
16 more
- 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
- 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
- 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
- 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 518
- 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
- 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
- 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
- 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
- 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
- 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
- 2023BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writingcited by 75
- 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
- 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
- 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
- 2021Machine Translation Verbosity Control for Automatic Dubbingcited by 18
- 2023Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understandingcited by 9
Built on by
Citing papers per year: 2025: 1.
1 paper.