mlsanityData to 7 Oct 2026

Discrete Action Space as a Prerequisite for GRPO Convergence in Small-Model Continuous Control

arXiv 2607.21626 ↗First posted 7 Jul 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

We study whether Group Relative Policy Optimization (GRPO) can fine-tune small language models for simulated quadrotor continuous-control tasks. In our benchmark, vanilla GRPO fine-tuning of Qwen-0.5B for 25 Hz quadrotor velocity control collapses to the trivial zero action: 0 percent success rate, with entropy falling from 0.35 to 0.03 within 60 steps. Two ablations - removing the jerk-penalty term and removing the KL anchor to the pretrained prior - each prevent entropy collapse, yet neither enables learning. When the action interface is replaced by a 5-way categorical choice over PID presets, training converges. The resulting controller traces a smoothness-reliability Pareto frontier along training duration; both endpoints are reported: 98.6 percent success with 0.656 m/s3 jerk at 64 steps, and 100 percent success with 1.103 m/s3 jerk, or 0.796 under a matched velocity cap, at 256 steps. The recipe is evaluated across three pretrained language models. As context, a re-tuned classical baseline, PID with Ki = 0.30 and vmax = 2.5, reaches the same 100 percent success rate at jerk 0.736 m/s3. A high-fidelity simulation using Crazyflie 2.1 dynamics surfaces a hover-region training-distribution gap.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 23 references that are in the map. First posted on arXiv 7 Jul 2026.

Builds on

15 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,297
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
2 more
  1. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  2. 2019Geometric Adaptive Control with Neural Networks for a Quadrotor UAV in Wind fieldscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Extending Group Relative Policy Optimization to Continuous Control: A Theoretical Framework for Robotic Reinforcement Learning
  2. 2025RAPTOR: A Foundation Policy for Quadrotor Control
  3. 2026Autonomous Reasoning for Spacecraft Control: A Large Language Model Framework with Group Relative Policy Optimization
  4. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization
  5. 2025Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments
  6. 2024Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight
  7. 2026GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents
  8. 2026GRPO Training Dynamics for Small Language Models