mlsanityData to 7 Oct 2026

GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learning

arXiv 2606.26917 ↗First posted 25 Jun 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Online reinforcement learning is widely used to align large language models (LLMs) with reward signals, yet training can be unstable under noisy or misspecified rewards. We identify a failure mode we call directional inconsistency: within a batch, a small set of high-reward rollouts induces representation-space preference directions that sharply disagree with the batch majority, resulting in high-variance and destabilizing updates. We propose geoalign, a lightweight plug-in for rollout curation in iterative policy optimization. Geoalign (i) forms within-prompt preference pairs, (ii) learns an online projector on per-rollout hidden states to concentrate reward-ordered displacement directions, and (iii) detects directionally inconsistent rollouts via their angular deviation from a batch consensus prototype and rectifies them with within-prompt stable alternatives. Geoalign is forward-pass only and adds negligible overhead. Across dialogue alignment with a learned reward model and mathematical reasoning with binary verified rewards, Geoalign improves final performance and reduces training oscillation, outperforming PF-PPO, PAR, PODS, and Seed-GRPO. These results suggest latent directional consensus as an effective reliability signal for online LLM RL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

32 of 36 references.

References · 32

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
20 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  4. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  5. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  7. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  8. 2025LIMR: Less is More for RL Scalingcited by 136
  9. 2024What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functionscited by 88
  10. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  11. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimizationcited by 87
  12. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
  13. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 75
  14. 2025Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoningcited by 49
  15. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 24
  16. 2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
  17. 2025Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Modelscited by 19
  18. 2025Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learningcited by 17
  19. 2025AlphaPO: Reward Shape Matters for LLM Alignmentcited by 14
  20. 2025A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learningcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
  2. 2026GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning
  3. 2025SR-GRPO: Stable Rank as an Intrinsic Geometric Reward for Large Language Model Alignment
  4. 2025COPO: Consistency-Aware Policy Optimization
  5. 2025RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
  6. 2025Online Distributionally Robust LLM Alignment via Regression to Relative Reward
  7. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learning
  8. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities