# Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities

arXiv: https://arxiv.org/abs/2505.08699 · HTML: https://mlsanity.com/paper/2505.08699/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2025.

## Built on by (most cited first, 27 of 27)

- [MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks](https://mlsanity.com/paper/2507.19634/) (2025)
- [Advancing Speech Understanding in Speech-Aware Language Models with GRPO](https://mlsanity.com/paper/2509.16990/) (2025)
- [HiKE: Hierarchical Evaluation Framework for Korean-English Code-Switching Speech Recognition](https://mlsanity.com/paper/2509.24613/) (2025)
- [Breaking Audio Large Language Models by Attacking Only the Encoder: A Universal Targeted Latent-Space Audio Attack](https://mlsanity.com/paper/2512.23881/) (2025)
- [Do LLM Decoders Listen Fairly? Benchmarking How Language Model Priors Shape Bias in Speech Recognition](https://mlsanity.com/paper/2604.21276/) (2026)
- [SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors](https://mlsanity.com/paper/2606.00460/) (2026)
- [Leveraging LLM-GNN Integration for Open-World Question Answering over Knowledge Graphs](https://mlsanity.com/paper/2604.13979/) (2026)
- [A Text-To-Text Alignment Algorithm for Better Evaluation of Modern Speech Recognition Systems](https://mlsanity.com/paper/2509.24478/) (2025)
- [Dynamic Multi-Expert Projectors with Stabilized Routing for Multilingual Speech Recognition](https://mlsanity.com/paper/2601.19451/) (2026)
- [Exploring the potential and limitations of Model Merging for Multi-Domain Adaptation in ASR](https://mlsanity.com/paper/2603.05354/) (2026)
- [In-Sync: Adaptation of Speech Aware Large Language Models for ASR with Word Level Timestamp Predictions](https://mlsanity.com/paper/2604.22817/) (2026)
- [From Text Metrics to Model Internals: A Study of Whisper ASR Hallucination Detection](https://mlsanity.com/paper/2606.23060/) (2026)
- [StanceBench: A Benchmark for Audio LLM-Based Interpersonal Stance Evaluation from Speech](https://mlsanity.com/paper/2607.22658/) (2026)
- [Reasoning Beyond Transcription: Audio Language Models on Child Stuttering Speech](https://mlsanity.com/paper/2609.07968/) (2026)
- [Measuring the Redundancy of Decoder Layers in SpeechLLMs](https://mlsanity.com/paper/2603.05121/) (2026)
- [AppTek Call-Center Dialogues: A Multi-Accent Long-Form Benchmark for English ASR](https://mlsanity.com/paper/2604.27543/) (2026)
- [Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment](https://mlsanity.com/paper/2605.00022/) (2026)
- [LEAF: Growing Trees Without Branching for Speech-Aware Large Language Model Post-Training](https://mlsanity.com/paper/2606.07610/) (2026)
- [ROMPAR: Morphological Completion and Demographic Unlearning for Romanian-Accented Speech Recognition](https://mlsanity.com/paper/2606.15984/) (2026)
- [Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleaving](https://mlsanity.com/paper/2607.01733/) (2026)
- [Towards Quantifying Benchmark Optimization in ASR Models](https://mlsanity.com/paper/2608.19936/) (2026)
- [Merging the Knowledge of LLMs for Automatic Speech Recognition](https://mlsanity.com/paper/2609.15743/) (2026)
- [Design of the IBM Granite 5.0 TurboCTC ASR Model](https://mlsanity.com/paper/2609.20104/) (2026)
- [AcoustiClaim: A Numeric Claim Benchmark with Instrument Ground Truth](https://mlsanity.com/paper/2609.30483/) (2026)
- [Audio Token Attention Is Predictable Before the Language Model Runs](https://mlsanity.com/paper/2609.38878/) (2026)
- [FFASR: Benchmarking Far-Field Automatic Speech Recognition using High-Fidelity Simulated RIRs](https://mlsanity.com/paper/2609.38897/) (2026)
- [Fairness Beyond a Single Run: Training-Seed Variability in Speech LLM Adaptation](https://mlsanity.com/paper/2609.38976/) (2026)

## Abstract (arXiv)

Granite-speech LLMs are compact and efficient speech language models specifically designed for English ASR and automatic speech translation (AST). The models were trained by modality aligning the 2B and 8B parameter variants of granite-3.3-instruct to speech on publicly available open-source corpora containing audio inputs and text targets consisting of either human transcripts for ASR or automatically generated translations for AST. Comprehensive benchmarking shows that on English ASR, which was our primary focus, they outperform several competitors' models that were trained on orders of magnitude more proprietary data, and they keep pace on English-to-X AST for major European languages, Japanese, and Chinese. The speech-specific components are: a conformer acoustic encoder using block attention and self-conditioning trained with connectionist temporal classification, a windowed query-transformer speech modality adapter used to do temporal downsampling of the acoustic embeddings and map them to the LLM text embedding space, and LoRA adapters to further fine-tune the text LLM. Granite-speech-3.3 operates in two modes: in speech mode, it performs ASR and AST by activating the encoder, projector, and LoRA adapters; in text mode, it calls the underlying granite-3.3-instruct model directly (without LoRA), essentially preserving all the text LLM capabilities and safety. Both models are freely available on HuggingFace (https://huggingface.co/ibm-granite/granite-speech-3.3-2b and https://huggingface.co/ibm-granite/granite-speech-3.3-8b) and can be used for both research and commercial purposes under a permissive Apache 2.0 license.