mlsanityData to 14 Oct 2026

Drop the Act: Probe-Filtered RL for Faithful Chain-of-Thought Reasoning

arXiv 2605.11467 ↗First posted 12 May 2026cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reasoning models post-hoc rationalize answers they have already committed to internally, producing chains of *reasoning theater*: deliberative-looking steps that contribute nothing to correctness. This wastes inference tokens, pollutes interpretability, and obscures what the model actually computed. We introduce **ProFIL** (**Pro**be-**Fil**tered Reinforcement Learning) to *reduce theater, increase chain-of-thought faithfulness, and shrink chain length* in a single, drop-in extension to Group Relative Policy Optimization (GRPO). A multi-head attention probe is trained *once* on the *frozen* base model to detect post-commitment steps from internal activations alone; during GRPO, rollouts whose probe score exceeds a threshold have their advantage zeroed. *Our central finding is that a probe trained on a frozen base, with verifier-derived labels and no human annotation, provides a stable signal that suppresses theater while resisting the RL-obfuscation failure mode predicted by prior work.* Across four reasoning domains (GSM8K, LiveCodeBench, ToolUse, MMLU-Redux) and two model architectures (Llama-8B, Qwen-7B), ProFIL reduces post-commitment theater by **11--100%**, raises faithful-fraction (e.g., +24pp on LiveCodeBench under an independent Claude 3.7 Sonnet judge), and shortens chains by 4--19%, all while preserving or improving task accuracy. ProFIL also beats a matched length-penalty GRPO baseline, isolating the gain as semantic commitment-detection rather than chain compression. Probe weights, training configurations, and rollouts are released across all four domains.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 23 of its 29 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

23 of 29 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025ConciseRL: Conciseness-Guided Reinforcement Learning for Efficient Reasoning Models22 May
  2. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models12 May
  3. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Models13 Apr
  4. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  5. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  6. 2023Let's Verify Step by Step31 May

Other references · 17

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  8. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,122
  9. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  10. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  11. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 703
  12. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
5 more
  1. 2025Reasoning Models Don't Always Say What They Thinkcited by 415
  2. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learningcited by 384
  3. 2023Faithful Chain-of-Thought Reasoningcited by 288
  4. 2024Are We Done with MMLU?cited by 194
  5. 2025RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?cited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Answer First, Reason Later: When Commitment Order Costs Accuracy in Diffusion Language Modelscited by 3
  2. 2026hacktrace: behavior-supervised detection of reward hacking during code generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dissecting Long-Chain-of-Thought Reasoning Models: An Empirical Study
  2. 2025Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
  3. 2026Self-Compression of Chain-of-Thought via Multi-Agent Reinforcement Learning
  4. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning
  5. 2025CTRLS: Chain-of-Thought Reasoning via Latent State-Transition
  6. 2025Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
  7. 2026ConMax: Confidence-Maximizing Compression for Efficient Chain-of-Thought Reasoning
  8. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought