mlsanityData to 7 Oct 2026

Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models

arXiv 2507.12428 ↗First posted 16 Jul 2025cs.CL cs.AI cs.LGcited by 16

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

also in 1 more
Abstract (arXiv)

Reasoning language models improve performance on complex tasks by generating long chains of thought (CoTs), but this process can also increase harmful outputs in adversarial settings. In this work, we ask whether the long CoTs can be leveraged for predictive safety monitoring: do the reasoning traces provide early signals of final response alignment that could enable timely intervention? We evaluate a range of monitoring methods using either CoT text or activations, including highly capable large language models, fine-tuned classifiers, and humans. First, we find that a simple linear probe trained on CoT activations significantly outperforms all text-based baselines in predicting whether a final response is safe or unsafe, with an average absolute increase of 13 in F1 scores over the best-performing alternatives. CoT texts are often unfaithful and misleading, while model latents provide a more reliable predictive signal. Second, the probe can be applied to early CoT segments before the response is generated, showing that alignment signals appear before reasoning completes. Error analysis reveals that the performance gap between text classifiers and the linear probe largely stems from a subset of responses we call performative CoTs, where the reasoning consistently contradicts the final response as the CoT progresses. Our findings generalize across model sizes, families, and safety benchmarks, suggesting that lightweight probes could enable real-time safety monitoring and early intervention during generation.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 37 of its 62 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

37 of 62 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors7 Jul
  2. 2025Thought Anchors: Which LLM Reasoning Steps Matter?23 Jun
  3. 2025Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion30 May
  4. 2025CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring29 May
  5. 2025Reasoning Models Don't Always Say What They Think8 May
  6. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation14 Mar
  7. 2025Chain-of-Thought Reasoning In The Wild Is Not Always Faithful11 Mar
  8. 2024Bias-Augmented Consistency Training Reduces Biased Reasoning in Chain-of-Thought8 Mar
  9. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning21 Feb
  10. 2023Measuring Faithfulness in Chain-of-Thought Reasoning17 Jul

Other references · 27

  1. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  2. 2024Qwen2.5 Technical Reportcited by 4,713
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2025s1: Simple test-time scalingcited by 1,311
  5. 2024OpenAI o1 System Cardcited by 1,220
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  7. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  9. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  10. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  11. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  12. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
15 more
  1. 2020Measuring Association Between Labels and Free-Text Rationalescited by 159
  2. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  3. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  4. 2025Persona Features Control Emergent Misalignmentcited by 76
  5. 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspectivecited by 75
  6. 2025Magistralcited by 57
  7. 2025Crosslingual Reasoning through Test-Time Scalingcited by 46
  8. 2025An Approach to Technical AGI Safety and Securitycited by 42
  9. 2025The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awarenesscited by 29
  10. 2024Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuningcited by 26
  11. 2025Measuring the Faithfulness of Thinking Drafts in Large Reasoning Modelscited by 22
  12. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 21
  13. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  14. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactionscited by 14
  15. 2025The Geometry of Self-Verification in a Task-Specific Reasoning Modelcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 9.

16 papers.

Sort:

By area:

  1. 2026Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thoughtcited by 35
  2. 2025Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?cited by 11
  3. 2025Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?cited by 11
  4. 2025HarmonyGuard: Toward Safety and Utility in Web Agents via Adaptive Policy Enhancement and Dual-Objective Optimizationcited by 10
  5. 2025MONICA: Real-Time Monitoring and Calibration of Chain-of-Thought Sycophancy in Large Reasoning Modelscited by 10
  6. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 9
  7. 2026Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoningcited by 7
  8. 2025The Impact of Off-Policy Training Data on Probe Generalisationcited by 6
  9. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  10. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 5
  11. 2026OLA: Output Language Alignment in Code-Switched LLM Interactionscited by 2
  12. 2026When is Your LLM Steerable?cited by 2
4 more
  1. 2026Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectoriescited by 1
  2. 2026Do Prompt-Elicited Trajectories Reflect Training-Time Reward Hacking? A Systematic Study on Monitoring Training-Time Reward Hacking in Code Generationcited by 0
  3. 2026When Do Model Internals Help? Exploring the Role of Representation Engineering in LLM Safetycited by 0
  4. 2026Caught in the Act: Probes Effectively Detect Sabotage and Catch Unverbalized Deceptioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Investigating CoT Monitorability in Large Reasoning Models
  2. 2026Monitoring the Internal Monologue: Probe Trajectories Reveal Reasoning Dynamics
  3. 2025Temporal Predictors of Outcome in Reasoning Language Models
  4. 2025AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models
  5. 2026The Fragility of Chain-of-Thought Monitoring Across Typologically Diverse Languages
  6. 2026When the Chain of Thought Knows Better: Failure Modes in Multi-Turn Reasoning Models
  7. 2026Detecting Hidden Chain-of-Thought in Large Language Models with Linguistic, Behavioral, and Mechanistic Indicators
  8. 2023Making Large Language Models Better Reasoners with Alignment