mlsanityData to 14 Oct 2026

N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization

arXiv 2606.10768 ↗First posted 9 Jun 2026cs.LG cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-of-distribution tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 27 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

21 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization9 Nov
  2. 2025Don't Tell the Answer, Truly Guide the Reasoning During RL Rollouts10 Oct
  3. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning13 Aug
  4. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar

Other references · 17

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2024OpenAI o1 System Cardcited by 1,244
  4. 2025Group Sequence Policy Optimizationcited by 835
  5. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  6. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 225
  7. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  8. 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 117
  9. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  10. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 85
  11. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 70
  12. 2025Soft Tokens, Hard Truthscited by 35
5 more
  1. 2025Hybrid Latent Reasoning via Reinforcement Learningcited by 32
  2. 2025Online Preference Alignment for Language Models via Count-based Explorationcited by 26
  3. 2015Measuring Word Significance using Distributed Representations of Wordscited by 21
  4. 2025SynAdapt: Learning Adaptive Reasoning in Large Language Models via Synthetic Continuous Chain-of-Thoughtcited by 13
  5. 2025Semantic Structure in Large Language Model Embeddingscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Markov Likelihood
  2. 2026Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
  3. 2025Geometric-Mean Policy Optimization
  4. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  5. 2025Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models
  6. 2026GRPO Training Dynamics for Small Language Models
  7. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models
  8. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statistic