mlsanityData to 13 Oct 2026

Kalman Filter Enhanced GRPO for Reinforcement Learning-Based Language Model Reasoning

arXiv 2505.07527 ↗First posted 12 May 2025cs.LGcited by 14

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The advantage function is a central concept in RL that helps reduce variance in policy gradient estimates. For language modeling, Group Relative Policy Optimization (GRPO) was proposed to use the within-group sample mean as a baseline for advantage normalization. This estimator can be sensitive to small group size and rollout-level stochasticity, which may lead to suboptimal advantage estimates in some settings. In this paper, we propose Kalman Filter Enhanced Group Relative Policy Optimization (KRPO), a lightweight variant that treats per-group rewards as noisy observations of a latent prompt-level reward baseline and uses a 1D Kalman filter to estimate both the baseline and its uncertainty. KRPO introduces no additional learned parameters and can be integrated into GRPO with minimal computational overhead. On mathematical reasoning benchmarks, KRPO consistently improves training reward curves and final accuracy over GRPO. These results suggest that adaptive advantage estimation is a promising direction for critic-free reinforcement learning in language model reasoning. The code is available at https://github.com/billhhh/KRPO_LLMs_RL.

Key facts. Cited by 14 papers in the map (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 32 references that are in the map. First posted on arXiv 12 May 2025.

Builds on

14 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2024KTO: Model Alignment as Prospect Theoretic Optimization2 Feb
  5. 2023Let's Verify Step by Step31 May
  6. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 8

  1. 2023GPT-4 Technical Reportcited by 18,278
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  4. 2015Continuous control with deep reinforcement learningcited by 3,290
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,185
  7. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  8. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 492

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025MAPO: Mixed Advantage Policy Optimizationcited by 11
  2. 2025λ-GRPO: Unifying the GRPO Frameworks with Learnable Token Preferencescited by 10
  3. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoningcited by 4
  4. 2026Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoningcited by 3
  5. 2025EQA-RM: A Generative Embodied Reward Model with Test-time Scalingcited by 2
  6. 2026Online Causal Kalman Filtering for Stable and Effective Policy Optimizationcited by 1
  7. 2026BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoningcited by 1
  8. 2025GAPO: Robust Advantage Estimation for Real-World Code LLMscited by 0
  9. 2026MHPO: Modulated Hazard-aware Policy Optimization for Stable Reinforcement Learningcited by 0
  10. 2026K-Score: Kalman Filter as a Principled Alternative to Reward Normalization in Reinforcement Learningcited by 0
  11. 2026CRPO: Character-centric Group Relative Policy Optimization for Role-aware Reasoning in Role-playing Agentscited by 0
  12. 2026BV-Blend: Uncertainty-Weighted Historical Baselines for Stable Critic-Free RL with Verifiable Rewardscited by 0
2 more
  1. 2026Revisiting On-policy Adversarial Black-Box Distillation: Calibrating Groupwise Reward Geometry for Effective Advantage Constructioncited by 0
  2. 2026When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025NGRPO: Negative-enhanced Group Relative Policy Optimization
  2. 2026Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation
  3. 2026Selective-Advantage Entropy-Adaptive Horizon GRPO: Asymmetric Token-Level Discounting for Efficient Reinforcement Learning of Language Models
  4. 2025AAPO: Enhancing the Reasoning Capabilities of LLMs with Advantage Margin
  5. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  6. 2026On the Hidden Objective Biases of Group-based Reinforcement Learning
  7. 2025EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
  8. 2026iGRPO: Self-Feedback-Driven LLM Reasoning