# ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models

arXiv: https://arxiv.org/abs/2505.17250 · HTML: https://mlsanity.com/paper/2505.17250/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 46 references that are in the map.
- First posted on arXiv 22 May 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 27 of 27)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs](https://mlsanity.com/paper/2412.21187/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models](https://mlsanity.com/paper/2503.16419/) (2025)
- [L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning](https://mlsanity.com/paper/2503.04697/) (2025)
- [Demystifying Long Chain-of-Thought Reasoning in LLMs](https://mlsanity.com/paper/2502.03373/) (2025)
- [O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruning](https://mlsanity.com/paper/2501.12570/) (2025)
- [Token-Budget-Aware LLM Reasoning](https://mlsanity.com/paper/2412.18547/) (2024)
- [Training Language Models to Reason Efficiently](https://mlsanity.com/paper/2502.04463/) (2025)
- [Compressed Chain of Thought: Efficient Reasoning Through Dense Representations](https://mlsanity.com/paper/2412.13171/) (2024)
- [ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning](https://mlsanity.com/paper/2504.01296/) (2025)
- [Mulberry: Empowering MLLM with o1-like Reasoning and Reflection via Collective Monte Carlo Tree Search](https://mlsanity.com/paper/2412.18319/) (2024)
- [Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems](https://mlsanity.com/paper/2412.09413/) (2024)
- [DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Models](https://mlsanity.com/paper/2503.04472/) (2025)
- [Concise Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2504.05185/) (2025)
- [Prompt-Based Length Controlled Generation with Reinforcement Learning](https://mlsanity.com/paper/2308.12030/) (2023)
- [Precise Length Control in Large Language Models](https://mlsanity.com/paper/2412.11937/) (2024)

## Built on by (most cited first, 14 of 14)

- [Mini-Omni-Reasoner: Token-Level Thinking-in-Speaking in Large Speech Models](https://mlsanity.com/paper/2508.15827/) (2025)
- [Train Long, Think Short: Curriculum Learning for Efficient Reasoning](https://mlsanity.com/paper/2508.08940/) (2025)
- [Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Models](https://mlsanity.com/paper/2601.11340/) (2026)
- [Multiplayer Nash Preference Optimization](https://mlsanity.com/paper/2509.23102/) (2025)
- [Learning to Reason Efficiently with Discounted Reinforcement Learning](https://mlsanity.com/paper/2510.23486/) (2025)
- [Contrastive On-Policy Distillation](https://mlsanity.com/paper/2607.19046/) (2026)
- [Adaptive Dual Reasoner: Large Reasoning Models Can Think Efficiently by Hybrid Reasoning](https://mlsanity.com/paper/2510.10207/) (2025)
- [TRiMS: Real-Time Tracking of Minimal Sufficient Length for Efficient Reasoning via RL](https://mlsanity.com/paper/2603.17449/) (2026)
- [Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2605.11467/) (2026)
- [BLUE: Toward Better Language Use in Efficient Vision-Language-Action Models for Autonomous Driving](https://mlsanity.com/paper/2606.08684/) (2026)
- [APR: Penalizing Structural Redundancy in Large Reasoning Models via Anchor-based Process Rewards](https://mlsanity.com/paper/2602.00760/) (2026)
- [WebClipper: Efficient Evolution of Web Agents with Graph-based Trajectory Pruning](https://mlsanity.com/paper/2602.12852/) (2026)
- [QLPO: Quadrant-weighted Sampling for Length-aware Policy Optimization](https://mlsanity.com/paper/2607.21793/) (2026)
- [AERA: Adaptive Evidence Residual Allocation for Efficient Test-Time Reasoning](https://mlsanity.com/paper/2608.27964/) (2026)

## Abstract (arXiv)

Large language models excel at complex tasks by breaking down problems into structured reasoning steps. However, reasoning traces often extend beyond reaching a correct answer, causing wasted computation, reduced readability, and hallucinations. To address this, we introduce a novel hyperparameter-free conciseness score used as a reward signal within a reinforcement learning framework to guide models toward generating correct and concise reasoning traces. This score is evaluated by a large language model acting as a judge, enabling dynamic, context-aware feedback beyond simple token length. Our method achieves state-of-the-art efficiency-accuracy trade-offs on the MATH dataset, reducing token usage by up to 31x on simple problems while improving accuracy by 7%, and on the hardest problems, it outperforms full reasoning by +7.5% accuracy with up to 3.6x fewer tokens. On TheoremQA, our method improves accuracy by +2.2% using 12.5x fewer tokens. We also conduct ablation studies on the judge model, reward composition, and problem difficulty, showing that our method dynamically adapts reasoning length based on problem difficulty and benefits significantly from stronger judges. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/ConciseRL.