# Advancing Speech Understanding in Speech-Aware Language Models with GRPO

arXiv: https://arxiv.org/abs/2509.16990 · HTML: https://mlsanity.com/paper/2509.16990/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 21 Sep 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 22 of 22)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Learning to Reason under Off-Policy Guidance](https://mlsanity.com/paper/2504.14945/) (2025)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2506.04779/) (2025)
- [On the Weaknesses of Reinforcement Learning for Neural Machine Translation](https://mlsanity.com/paper/1907.01752/) (2019)
- [Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering](https://mlsanity.com/paper/2503.11197/) (2025)
- [Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?](https://mlsanity.com/paper/2505.09439/) (2025)
- [Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities](https://mlsanity.com/paper/2505.08699/) (2025)
- [BLEUBERI: BLEU is a surprisingly effective reward for instruction following](https://mlsanity.com/paper/2505.11080/) (2025)
- [LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Models](https://mlsanity.com/paper/2308.10390/) (2023)
- [Slamming: Training a Speech Language Model on One GPU in a Day](https://mlsanity.com/paper/2502.15814/) (2025)
- [Self-Improvement for Audio Large Language Model using Unlabeled Speech](https://mlsanity.com/paper/2507.20169/) (2025)

## Built on by (most cited first, 4 of 4)

- [SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors](https://mlsanity.com/paper/2606.00460/) (2026)
- [A First-Order Logic-Based Alternative to Reward Models in RLHF](https://mlsanity.com/paper/2512.14100/) (2025)
- [LEAF: Growing Trees Without Branching for Speech-Aware Large Language Model Post-Training](https://mlsanity.com/paper/2606.07610/) (2026)
- [Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translation](https://mlsanity.com/paper/2609.26536/) (2026)

## Abstract (arXiv)

In this paper, we introduce a Group Relative Policy Optimization (GRPO)-based method for training Speech-Aware Large Language Models (SALLMs) on open-format speech understanding tasks, such as Spoken Question Answering and Automatic Speech Translation. SALLMs have proven highly effective for speech understanding tasks. GRPO has recently gained traction for its efficiency in training LLMs, and prior work has explored its application to SALLMs, primarily in multiple-choice tasks. Building on this, we focus on open-format tasks that better reflect the generative abilities of the models. Our approach leverages GRPO with BLEU as the reward signal to optimize SALLMs, and we demonstrate empirically that it surpasses standard SFT across several key metrics. Finally, we explore the potential of incorporating off-policy samples within GRPO for these tasks, highlighting avenues for further improvement and further research.