mlsanityData to 14 Oct 2026

Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models

arXiv 2605.17770 ↗First posted 18 May 2026cs.AI cs.CLcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

The advancement of Large Reasoning Models (LRMs) has catalyzed a paradigm shift from reactive ``fast thinking'' text generation to systematic, step-by-step ``slow thinking'' reasoning, unlocking state-of-the-art performance in complex mathematical and logical tasks. However, the field faces textitthe fundamental gap between token-level behavioral analysis and internal reasoning mechanisms, and the instability of reinforcement learning (RL) for reasoning optimization relying on costly external verifiers. We identify and formally define Entropy-Gradient Inversion, a robust negative correlation between token entropy and logit gradients that acts as a definitive geometric fingerprint for LRM reasoning capability. Building on this, we propose Correlation-Regularized Group Policy Optimization (CorR-PO), which embeds this inversion signature into RL reward regularization. Extensive experiments on various reasoning benchmarks across multiple model scales show CorR-PO consistently outperforms state-of-the-art baselines, confirming that stronger inversion directly correlates with superior reasoning performance.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 43 of its 62 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

43 of 62 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning17 Dec
  2. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models5 Jul
  3. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  5. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  6. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  8. 2023Let's Verify Step by Step31 May
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
  11. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2025Qwen3 Technical Reportcited by 8,820
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
20 more
  1. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  3. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  4. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  5. 2018Predictive Uncertainty Estimation via Prior Networkscited by 557
  6. 2021Causal Abstractions of Neural Networkscited by 422
  7. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  8. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  9. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 304
  10. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  11. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  12. 2025Demystifying Reasoning Dynamics with Mutual Information: Thinking Tokens are Information Peaks in LLM Reasoningcited by 51
  13. 2026AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Securitycited by 50
  14. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 45
  15. 2024Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inferencecited by 29
  16. 2024What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspectivecited by 29
  17. 2026The Why Behind the Action: Unveiling Internal Drivers via Agentic Attributioncited by 11
  18. 2026Truth as a Trajectory: What Internal Representations Reveal About Large Language Model Reasoningcited by 10
  19. 2025RCP-Merging: Merging Long Chain-of-Thought Models with Domain-Specific Models by Considering Reasoning Capability as Priorcited by 5
  20. 2026ReasonAny: Incorporating Reasoning Capability to Any Model via Simple and Effective Model Mergingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026SFT Leaves a Reproducible Signature in Activation Geometry: Calibrating Post-Training Interpretability Against Run-to-Run Noisecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ERPO: Token-Level Entropy-Regulated Policy Optimization for Large Reasoning Models
  2. 2025Revisiting Entropy in Reinforcement Learning for Large Reasoning Models
  3. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  4. 2025Rethinking Entropy Regularization in Large Reasoning Models
  5. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs
  6. 2026ETR: Entropy Trend Reward for Efficient Chain-of-Thought Reasoning
  7. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  8. 2025Efficient Reinforcement Learning with Semantic and Token Entropy for LLM Reasoning