mlsanityData to 14 Oct 2026

Rethinking the Comparison Unit in Sequence-Level Reinforcement Learning: An Equal-Length Paired Training Framework from Loss Correction to Sample Construction

arXiv 2604.17328 ↗First posted 19 Apr 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

This paper investigates the length problem in sequence-level relative reinforcement learning. We observe that, although existing methods partially alleviate length-related phenomena, a more fundamental issue remains insufficiently characterized: the comparison units used during training lack inherent comparability. Building on this observation, we propose a new perspective: the length problem should not be viewed merely as a loss-scaling or normalization bias, but rather as a comparison unit construction problem. We further establish a sample-construction-based training framework that, instead of applying post-hoc corrections to unequal-length responses, proactively constructs equal-length, alignable, and comparable training segments during generation. Within this framework, we propose EqLen, a concrete method applicable to group-relative comparison algorithms such as GRPO, GSPO, and RLOO. Through dual-track synchronous generation, prefix inheritance, and segment masking, EqLen efficiently collects effective equal-length training segments and enables stable

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 24 of its 33 references that are in the map. First posted on arXiv 19 Apr 2026.

Builds on

24 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Tackling Length Inflation Without Trade-offs: Group Relative Reward Rescaling for Reinforcement Learning11 Mar
  2. 2025DLER: Doing Length pEnalty Right - Incentivizing More Intelligence per Token via Reinforcement Learning16 Oct

Other references · 22

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  11. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  12. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 280
10 more
  1. 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
  2. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  3. 2025TreeRPO: Tree Relative Policy Optimizationcited by 46
  4. 2026VESPO: Variational Sequence-Level Soft Policy Optimization for Stable Off-Policy LLM Trainingcited by 11
  5. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoningcited by 9
  6. 2026Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVRcited by 8
  7. 2025Bias Fitting to Mitigate Length Bias of Reward Model in RLHFcited by 7
  8. 2025Disentangling Length Bias In Preference Learning Via Response-Conditioned Modelingcited by 6
  9. 2025Mitigating Length Bias in RLHF through a Causal Lenscited by 6
  10. 2026Boosting LLM Reasoning via Human-Inspired Reward Shapingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  2. 2024Disentangling Length from Quality in Direct Preference Optimization
  3. 2026On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
  4. 2026On the Optimal Reasoning Length for RL-Trained Language Models
  5. 2026Your Group-Relative Advantage Is Biased
  6. 2026A Relative-Budget Theory for Reinforcement Learning with Verifiable Rewards in Large Language Model Reasoning
  7. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  8. 2026From Absolute to Relative: Rethinking Reward Shaping in Group-Based Reinforcement Learning