# Beyond the Trade-off: Self-Supervised Reinforcement Learning for Reasoning Models' Instruction Following

arXiv: https://arxiv.org/abs/2508.02150 · HTML: https://mlsanity.com/paper/2508.02150/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 33 of its 42 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 30 of 33)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [FOLIO: Natural Language Reasoning with First-Order Logic](https://mlsanity.com/paper/2209.00840/) (2022)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)
- [Seed1.5-Thinking: Advancing Superb Reasoning Models with Reinforcement Learning](https://mlsanity.com/paper/2504.13914/) (2025)
- [BIG-Bench Extra Hard](https://mlsanity.com/paper/2502.19187/) (2025)
- [FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Models](https://mlsanity.com/paper/2310.20410/) (2023)
- [WritingBench: A Comprehensive Benchmark for Generative Writing](https://mlsanity.com/paper/2503.05244/) (2025)
- [Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models](https://mlsanity.com/paper/2406.13542/) (2024)
- [COLLIE: Systematic Construction of Constrained Text Generation Tasks](https://mlsanity.com/paper/2307.08689/) (2023)
- [CFBench: A Comprehensive Constraints-Following Benchmark for LLMs](https://mlsanity.com/paper/2408.01122/) (2024)
- [Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Models](https://mlsanity.com/paper/2506.11116/) (2025)
- [AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios](https://mlsanity.com/paper/2505.16944/) (2025)
- [VerIF: Verification Engineering for Reinforcement Learning in Instruction Following](https://mlsanity.com/paper/2506.09942/) (2025)
- [Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Models](https://mlsanity.com/paper/2404.02823/) (2024)
- [From Complex to Simple: Enhancing Multi-Constraint Complex Instruction Following Ability of Large Language Models](https://mlsanity.com/paper/2404.15846/) (2024)
- [SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Models](https://mlsanity.com/paper/2412.11605/) (2024)
- [SciKnowEval: Evaluating Multi-level Scientific Knowledge of Large Language Models](https://mlsanity.com/paper/2406.09098/) (2024)
- [UltraIF: Advancing Instruction Following from the Wild](https://mlsanity.com/paper/2502.04153/) (2025)
- [Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models](https://mlsanity.com/paper/2506.01413/) (2025)
- [Stronger Models are NOT Stronger Teachers for Instruction Tuning](https://mlsanity.com/paper/2411.07133/) (2024)

## Built on by (most cited first, 3 of 3)

- [AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios](https://mlsanity.com/paper/2601.20613/) (2026)
- [Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence](https://mlsanity.com/paper/2608.31075/) (2026)
- [Do Thinking Tokens Help with Safety?](https://mlsanity.com/paper/2606.25013/) (2026)

## Abstract (arXiv)

Reasoning models excel in complex problem solving but exhibit a concerning trade off between reasoning capabilities and instruction following abilities. Existing approaches for improving instruction following rely on stronger external models, creating methodological bottlenecks and practical limitations including increased costs and accessibility constraints. We propose a self-supervised RL framework that leverages reasoning models' own internal signals to improve instruction following capabilities without external supervision. Extensive experiments demonstrate that our framework significantly improves instruction following capabilities while maintaining reasoning performance, offering a scalable and cost-effective approach to enhance instruction following in reasoning models. The data and code are publicly available at https://github.com/Rainier-rq/verl-if.