mlsanityData to 14 Oct 2026

Advancing Speech Understanding in Speech-Aware Language Models with GRPO

arXiv 2509.16990 ↗First posted 21 Sep 2025cs.CL cs.AI cs.LGcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

In this paper, we introduce a Group Relative Policy Optimization (GRPO)-based method for training Speech-Aware Large Language Models (SALLMs) on open-format speech understanding tasks, such as Spoken Question Answering and Automatic Speech Translation. SALLMs have proven highly effective for speech understanding tasks. GRPO has recently gained traction for its efficiency in training LLMs, and prior work has explored its application to SALLMs, primarily in multiple-choice tasks. Building on this, we focus on open-format tasks that better reflect the generative abilities of the models. Our approach leverages GRPO with BLEU as the reward signal to optimize SALLMs, and we demonstrate empirically that it surpasses standard SFT across several key metrics. Finally, we explore the potential of incorporating off-policy samples within GRPO for these tasks, highlighting avenues for further improvement and further research.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 28 references that are in the map. First posted on arXiv 21 Sep 2025.

Builds on

22 of 28 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Learning to Reason under Off-Policy Guidance21 Apr
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2017Decoupled Weight Decay Regularizationcited by 1,831
  5. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  6. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 521
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  8. 2024MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmarkcited by 288
  9. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  10. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmarkcited by 103
  11. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  12. 2025Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answeringcited by 65
6 more
  1. 2025Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?cited by 46
  2. 2025Granite-speech: open-source speech-aware LLMs with strong English ASR capabilitiescited by 27
  3. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  4. 2023LibriSQA: A Novel Dataset and Framework for Spoken Question Answering with Large Language Modelscited by 14
  5. 2025Slamming: Training a Speech Language Model on One GPU in a Daycited by 12
  6. 2025Self-Improvement for Audio Large Language Model using Unlabeled Speechcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectorscited by 4
  2. 2025A First-Order Logic-Based Alternative to Reward Models in RLHFcited by 0
  3. 2026LEAF: Growing Trees Without Branching for Speech-Aware Large Language Model Post-Trainingcited by 0
  4. 2026Transcribe, Translate, and Optimize: Joint Reward Learning for Speech Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GRPO Training Dynamics for Small Language Models
  2. 2026Transformation-Augmented GRPO for Enhancing Exploration in Reasoning of Large Language Models
  3. 2025Composing Policy Gradients and Prompt Optimization for Language Model Programs
  4. 2026GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
  5. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models
  6. 2025Multi-Reward GRPO for Stable and Prosodic Single-Codebook TTS LLMs at Scale
  7. 2025VAPO: End-to-end Slide-Enhanced Speech Recognition with Omni-modal Large Language Models
  8. 2025Align2Speak: Improving TTS for Low Resource Languages via ASR-Guided Online Preference Optimization