# SparQLe: Speech Queries to Text Translation Through LLMs

arXiv: https://arxiv.org/abs/2502.09284 · HTML: https://mlsanity.com/paper/2502.09284/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 38 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Self-training and Pre-training are Complementary for Speech Recognition](https://mlsanity.com/paper/2010.11430/) (2020)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [ESPnet-ST: All-in-One Speech Translation Toolkit](https://mlsanity.com/paper/2004.10234/) (2020)
- [Connecting Speech Encoder and Large Language Model for ASR](https://mlsanity.com/paper/2309.13963/) (2023)
- [BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing](https://mlsanity.com/paper/2309.00916/) (2023)
- [SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation](https://mlsanity.com/paper/2310.09424/) (2023)
- [SLM: Bridge the thin gap between speech and text foundation models](https://mlsanity.com/paper/2310.00230/) (2023)
- [SpeechVerse: A Large-scale Generalizable Audio Language Model](https://mlsanity.com/paper/2405.08295/) (2024)
- [Machine Translation Verbosity Control for Automatic Dubbing](https://mlsanity.com/paper/2110.03847/) (2021)
- [Speech-to-Text Adapter and Speech-to-Entity Retriever Augmented LLMs for Speech Understanding](https://mlsanity.com/paper/2306.07944/) (2023)

## Built on by (most cited first, 1 of 1)

- [SPIRIT: Patching Speech Language Models against Jailbreak Attacks](https://mlsanity.com/paper/2505.13541/) (2025)

## Abstract (arXiv)

With the growing influence of Large Language Models (LLMs), there is increasing interest in integrating speech representations with them to enable more seamless multi-modal processing and speech understanding. This study introduces a novel approach that combines self-supervised speech representations with instruction-tuned LLMs for speech-to-text translation. The proposed approach leverages a modality adapter to align extracted speech features with instruction-tuned LLMs using English speech data. Our experiments demonstrate that this method effectively preserves the semantic content of the input speech and serves as an effective bridge between self-supervised speech models and instruction-tuned LLMs, offering a promising approach for various speech understanding applications.