# Aligned, Orthogonal or In-conflict: When can we safely optimize Chain-of-Thought?

arXiv: https://arxiv.org/abs/2603.30036 · HTML: https://mlsanity.com/paper/2603.30036/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 16 of its 17 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 16 of 16)

- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Training Large Language Models to Reason in a Continuous Latent Space](https://mlsanity.com/paper/2412.06769/) (2024)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation](https://mlsanity.com/paper/2503.11926/) (2025)
- [Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety](https://mlsanity.com/paper/2507.11473/) (2025)
- [Thought Anchors: Which LLM Reasoning Steps Matter?](https://mlsanity.com/paper/2506.19143/) (2025)
- [LlamaFirewall: An open source guardrail system for building secure AI agents](https://mlsanity.com/paper/2505.03574/) (2025)
- [When Chain of Thought is Necessary, Language Models Struggle to Evade Monitors](https://mlsanity.com/paper/2507.05246/) (2025)
- [Stress Testing Deliberative Alignment for Anti-Scheming Training](https://mlsanity.com/paper/2509.15541/) (2025)
- [Monitoring Monitorability](https://mlsanity.com/paper/2512.18311/) (2025)
- [A Survey of In-Context Reinforcement Learning](https://mlsanity.com/paper/2502.07978/) (2025)
- [Large language models can learn and generalize steganographic chain-of-thought under process supervision](https://mlsanity.com/paper/2506.01926/) (2025)
- [CAPO: Towards Enhancing LLM Reasoning through Generative Credit Assignment](https://mlsanity.com/paper/2508.02298/) (2025)
- [A Pragmatic Way to Measure Chain-of-Thought Monitorability](https://mlsanity.com/paper/2510.23966/) (2025)
- [Reasoning Under Pressure: How do Training Incentives Influence Chain-of-Thought Monitorability?](https://mlsanity.com/paper/2512.00218/) (2025)
- [Quantifying the Necessity of Chain of Thought through Opaque Serial Depth](https://mlsanity.com/paper/2603.09786/) (2026)

## Built on by (most cited first, 5 of 5)

- [Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning](https://mlsanity.com/paper/2605.24286/) (2026)
- [Length Penalties Make Chain-of-Thought Less Monitorable](https://mlsanity.com/paper/2607.09786/) (2026)
- [Hidden Reasoning Must Leak, but Need Not Be Readable: Fundamental Opportunities and Limits for Chain-of-Thought Monitoring](https://mlsanity.com/paper/2609.37312/) (2026)
- [A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control](https://mlsanity.com/paper/2610.03458/) (2026)
- [Efficient Reasoning Training Does Not Always Harm CoT Faithfulness and Monitorability](https://mlsanity.com/paper/2610.03509/) (2026)

## Abstract (arXiv)

Chain-of-Thought (CoT) monitoring, in which automated systems monitor the CoT of an LLM, is a promising approach for effectively overseeing AI systems. However, the extent to which a model's CoT helps us oversee the model - the monitorability of the CoT - can be affected by training, for instance by the model learning to hide important features of its reasoning. We propose and empirically validate a conceptual framework for predicting when and why this occurs. We model LLM post-training as an RL environment where the reward decomposes into two terms: one term depending on final outputs and another term depending on the CoT. Our framework allows us to classify these two terms as "aligned", "orthogonal", or "in-conflict" before training. We predict that training with in-conflict terms will reduce monitorability, orthogonal terms will not affect it, and aligned terms will improve it. To validate our framework, we use it to classify a set of RL environments, train LLMs within those environments, and evaluate how training affects CoT monitorability. We find that (1) training with "in-conflict" reward terms reduces CoT monitorability and (2) optimizing in-conflict reward terms is difficult.