# AlignFormer: Modality Matching Can Achieve Better Zero-shot Instruction-Following Speech-LLM

arXiv: https://arxiv.org/abs/2412.01145 · HTML: https://mlsanity.com/paper/2412.01145/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2024.

## Built on by (most cited first, 12 of 12)

- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Optimal Transport Regularization for Speech Text Alignment in Spoken Language Models](https://mlsanity.com/paper/2508.08131/) (2025)
- [Language Family Matters: Evaluating LLM-Based ASR Across Linguistic Boundaries](https://mlsanity.com/paper/2601.18899/) (2026)
- [ALARM: Audio-Language Alignment for Reasoning Models](https://mlsanity.com/paper/2603.09556/) (2026)
- [Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMs](https://mlsanity.com/paper/2606.09366/) (2026)
- [FastSLM: Hierarchical Temporal Abstraction for Efficient Long-Form Speech Adaptation](https://mlsanity.com/paper/2601.06199/) (2026)
- [Decoder-only Conformer with Modality-aware Sparse Mixtures of Experts for ASR](https://mlsanity.com/paper/2602.12546/) (2026)
- [Speak in Context: Multilingual ASR with Speech Context Alignment via Contrastive Learning](https://mlsanity.com/paper/2603.06505/) (2026)
- [TASU2: Controllable CTC Simulation for Alignment and Low-Resource Adaptation of Speech LLMs](https://mlsanity.com/paper/2604.08384/) (2026)
- [Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving](https://mlsanity.com/paper/2607.01733/) (2026)
- [When Does a Spoken Agent Have Enough Evidence to Act? The PACT-SLM Contract Test](https://mlsanity.com/paper/2609.38232/) (2026)
- [DirectSpeech2LLM: A Simple End-to-End Framework to Mitigate Prompt Overfitting in Speech-LLMs](https://mlsanity.com/paper/2610.08085/) (2026)

## Abstract (arXiv)

Integrating speech into LLM (speech-LLM) has gaining increased attention recently. The mainstream solution is to connect a well-trained speech encoder and LLM with a neural adapter. However, the length mismatch between the speech and text sequences are not well handled, leading to imperfect modality matching between the speech and text. In this work, we propose a novel neural adapter, AlignFormer, to reduce the length gap between the two modalities. AlignFormer consists of CTC and dynamic-window QFormer layers, where the CTC alignment provides the dynamic window information for QFormer. The LLM backbone is frozen in training to preserve its text capability, especially the instruction following capability. When training with ASR data only, the proposed AlignFormer unlocks the instruction following capability for speech-LLM and the model can perform zero-shot speech translation (ST) and speech question answering (SQA) tasks. In fact, speech-LLM with AlignFormer can theoretically perform any tasks that the LLM backbone can deal with in the speech version. To evaluate the effectiveness of the instruction-following speech-LLM, we propose to use instruction following rate (IFR) and offer a systematic perspective for the IFR evaluation. In addition, we find that the audio position in training would affect the instruction following capability of speech-LLM and conduct an in-depth study on it. Our findings show that audio-first training achieves higher IFR than instruction-first training. The AlignFormer can achieve a near 100% IFR with audio-first training and game-changing improvements from zero to non-zero IFR on some evaluation data with instruction-first training. We believe that this study is a big step towards the perfect speech and text modality matching in the LLM embedding space.