Transferable speech-to-text large language model alignment module
On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods
Abstract (arXiv)
By leveraging the power of Large Language Models(LLMs) and speech foundation models, state of the art speech-text bimodal works can achieve challenging tasks like spoken translation(ST) and question answering(SQA) altogether with much simpler architectures. In this paper, we utilize the capability of Whisper encoder and pre-trained Yi-6B. Empirical results reveal that modal alignment can be achieved with one layer module and hundred hours of speech-text multitask corpus. We further swap the Yi-6B with human preferences aligned version of Yi-6B-Chat during inference, and discover that the alignment capability is applicable as well. In addition, the alignment subspace revealed by singular value decomposition(SVD) also implies linear alignment subspace is sparse, which leaves the possibility to concatenate other features like voice-print or video to expand modality.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 20 references that are in the map. First posted on arXiv 19 Jun 2024.
Builds on
20 of 20 references.
Closest ancestors · 1
Same direction, newest first
Other references · 19
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2023Visual Instruction Tuningcited by 7,869
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
- 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
7 more
- 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
- 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
- 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 197
- 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
- 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
- 2018AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scalecited by 128
- 2023LLaSM: Large Language and Speech Modelcited by 58
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation
- 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models
- 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Models
- 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
- 2025Aligning Multimodal LLM with Human Preference: A Survey
- 2024AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability
- 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
- 2026Alignment-Free Text-Audiobox for Voice Dubbing and Full-Duplex Dialogue Synthesis