mlsanityData to 14 Oct 2026

Learning to Reason with Mixture of Tokens

arXiv 2509.21482 ↗First posted 25 Sep 2025cs.CL cs.AI cs.LGcited by 3

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards (RLVR) has become a leading approach for improving large language model (LLM) reasoning capabilities. Most current methods follow variants of Group Relative Policy Optimization, which samples multiple reasoning completions, scores them relative to each other, and adjusts the policy accordingly. However, these approaches invariably sample discrete tokens at each reasoning step, discarding the rich distributional information in the model's probability distribution over candidate tokens. While preserving and utilizing this distributional information has proven beneficial in non-RL settings, current RLVR methods seem to be unnecessarily constraining the reasoning search space by not using this information. To address this limitation, we investigate mixture-of-token generation (MoT-G) in RLVR. We present a unified framework that generalizes existing MoT-G approaches, including existing training-free methods that construct mixture embeddings as weighted sums over token embeddings, and extend RLVR to operate directly in this continuous mixture space for generating chain-of-thought. Evaluating two MoT-G variants on Reasoning-Gym, a suite of reasoning-intensive language tasks, we find that MoT--G methods achieve substantial improvements (5--35 % gains on 7 out of 10 tasks) compared to standard decoding with the Qwen2.5-1.5B model, while reaching comparable accuracy with half the number of trajectories, suggesting improved training efficiency. Through comprehensive hidden-state and token-level analyses, we provide evidence that MoT--G's benefits may stem from its ability to maintain higher hidden-state entropy throughout the reasoning process and promote exploration in token space.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 32 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

25 of 32 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  4. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar

Other references · 19

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Group Sequence Policy Optimizationcited by 835
  5. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  6. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  7. 2025The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexitycited by 302
  8. 2024From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Stepcited by 235
  9. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 130
  10. 2025Soft Thinking: Unlocking the Reasoning Potential of LLMs in Continuous Concept Spacecited by 117
  11. 2025SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMscited by 116
  12. 2025Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoningcited by 91
7 more
  1. 2025Winning Gold at IMO 2025 with a Model-Agnostic Verification-and-Refinement Pipelinecited by 82
  2. 2025Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thoughtcited by 77
  3. 2025Soft Tokens, Hard Truthscited by 35
  4. 2025AlphaOne: Reasoning Models Thinking Slow and Fast at Test Timecited by 29
  5. 2025Text Generation Beyond Discrete Token Samplingcited by 21
  6. 2024HDFlow: Enhancing LLM Complex Problem-Solving with Hybrid Thinking and Dynamic Workflowscited by 13
  7. 2025Learning Refined Document Representations for Dense Retrieval via Deliberate Thinkingcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimizationcited by 21
  2. 2026OPPO: Bayesian Value Recursion for Token-Level Credit Assignment in LLM Reasoningcited by 4
  3. 2026SuperThoughts: Reasoning Tokens in Superpositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models
  2. 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
  3. 2026DARL: Encouraging Diverse Answers for General Reasoning without Verifiers
  4. 2026GMTS: Gradient Magnitude-based Token Selection Improves RLVR Training for LLM Reasoning
  5. 2026To Mix or To Merge: Toward Multi-Domain Reinforcement Learning for Large Language Models
  6. 2026Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs
  7. 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
  8. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs