mlsanityData to 14 Oct 2026

SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimization

arXiv 2511.06411 ↗First posted 9 Nov 2025cs.AI cs.LGcited by 21

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The soft-thinking paradigm for Large Language Model (LLM) reasoning can outperform the conventional discrete-token Chain-of-Thought (CoT) reasoning in some scenarios, underscoring its research and application value. However, while the discrete-token CoT reasoning pattern can be reinforced through policy optimization algorithms such as group relative policy optimization (GRPO), extending the soft-thinking pattern with Reinforcement Learning (RL) remains challenging. This difficulty stems from the complexities of injecting stochasticity into soft-thinking tokens and updating soft-thinking policies accordingly. As a result, previous attempts to combine soft-thinking with GRPO typically underperform their discrete-token GRPO counterparts. To fully unlock the potential of soft-thinking, this paper presents a novel policy optimization algorithm, SofT-GRPO, to reinforce LLMs under the soft-thinking reasoning pattern. SofT-GRPO injects the Gumbel noise into logits, employs the Gumbel-Softmax technique to avoid soft-thinking tokens outside the pre-trained embedding space, and leverages the reparameterization trick in policy gradient. We conduct experiments across base LLMs ranging from 1.5B to 7B parameters, and results demonstrate that SofT-GRPO enables soft-thinking LLMs to slightly outperform discrete-token GRPO on Pass@1 (+0.13% on average accuracy), while exhibiting a substantial uplift on Pass@32 (+2.19% on average accuracy). Codes and weights are available on https://github.com/zz1358m/SofT-GRPO-master

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 55 references that are in the map. First posted on arXiv 9 Nov 2025.

Builds on

44 of 55 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  2. 2025SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training9 Oct
  3. 2025Learning to Reason with Mixture of Tokens25 Sep
  4. 2025Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning11 Aug
  5. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2025Training Language Models to Reason Efficiently6 Feb
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  10. 2024Reinforcement Learning Enhanced LLMs: A Survey5 Dec
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 33

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  4. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  6. 2024DeepSeek-V3 Technical Reportcited by 811
  7. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  8. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  9. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  10. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 263
  11. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 225
  12. 2024Compressed Chain of Thought: Efficient Reasoning Through Dense Representationscited by 201
21 more
  1. 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 117
  2. 2025LightThinker: Thinking Step-by-Step Compressioncited by 104
  3. 2025Efficient Reasoning Models: A Surveycited by 95
  4. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 91
  5. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  6. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  7. 2025Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chainscited by 85
  8. 2025A Survey on Latent Reasoningcited by 81
  9. 2025Reasoning Beyond Language: A Comprehensive Survey on Latent Chain-of-Thought Reasoningcited by 78
  10. 2025SIM-CoT: Supervised Implicit Chain-of-Thoughtcited by 70
  11. 2025Defeating the Training-Inference Mismatch via FP16cited by 66
  12. 2025SoftCoT++: Test-Time Scaling with Soft Chain-of-Thought Reasoningcited by 41
  13. 2025LLM Latent Reasoning as Chain of Superpositioncited by 39
  14. 2025LLMs are Single-threaded Reasoners: Demystifying the Working Mechanism of Soft Thinkingcited by 38
  15. 2025Soft Tokens, Hard Truthscited by 35
  16. 2025Optimizing Anytime Reasoning via Budget Relative Policy Optimizationcited by 27
  17. 2024Are More LLM Calls All You Need? Towards Scaling Laws of Compound Inference Systemscited by 25
  18. 2025Text Generation Beyond Discrete Token Samplingcited by 21
  19. 2025Beyond the Sampled Token: Preserving Candidate Support in RLVRcited by 18
  20. 2025SynAdapt: Learning Adaptive Reasoning in Large Language Models via Synthetic Continuous Chain-of-Thoughtcited by 13
  21. 2025Reasoning-CV: Fine-tuning Powerful Reasoning LLMs for Knowledge-Assisted Claim Verificationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 21.

21 papers.

Sort:

By area:

  1. 2026The Latent Space: Foundation, Evolution, Mechanism, Ability, and Outlookcited by 43
  2. 2026Latent-GRPO: Group Relative Policy Optimization for Latent Reasoningcited by 10
  3. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learningcited by 7
  4. 2026LEPO: Latent Reasoning Policy Optimization for Large Language Modelscited by 4
  5. 2026Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirementscited by 3
  6. 2026Demystifying Hidden-State Recurrence: Switchable Latent Reasoning with On-Policy Reinforcement Learningcited by 2
  7. 2026MUX: Continuous Reasoning via Multiplexed Tokenscited by 2
  8. 2026Learning Concepts, Not Tokens: Self-Supervised Semantic Alignment for Language Modelscited by 1
  9. 2026Latent Thought Credit: Multi-Answer Credit Assignment for Latent Reasoningcited by 1
  10. 2026Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Mergingcited by 1
  11. 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPOcited by 1
  12. 2026RuPLaR : Efficient Latent Compression of LLM Reasoning Chains with Rule-Based Priors From Multi-Step to One-Stepcited by 0
9 more
  1. 2026Value-Gradient Hypothesis of RL for LLMscited by 0
  2. 2026TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimizationcited by 0
  3. 2026One Token per Multimodal Evidence: Latent Memory for Resource-Constrained QAcited by 0
  4. 2026N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimizationcited by 0
  5. 2026The Weight of Silence: A Causal Case for Weights Over the Scratchpad in Latent Chess Reasoningcited by 0
  6. 2026A Model with No Head and Many Thoughtscited by 0
  7. 2026The Dynamics of Continuous Mixture Collapse in Language Modelscited by 0
  8. 2026Soft Spatial Reasoningcited by 0
  9. 2026Beyond Policy Rollouts: Embedding-Space Directional Exploration with Anchored Noise for LLM RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Soft Adaptive Policy Optimization
  2. 2026Soft Sequence Policy Optimization
  3. 2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoning
  4. 2026Token-Level Policy Optimization: Linking Group-Level Rewards to Token-Level Aggregation via Sequence-Level Likelihood
  5. 2026Long Chain-of-Thought Compression via Fine-Grained Group Policy Optimization
  6. 2025GTPO and GRPO-S: Token and Sequence-Level Reward Shaping with Policy Entropy
  7. 2026CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
  8. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning