# PA3: Policy-Aware Agent Alignment through Chain-of-Thought

arXiv: https://arxiv.org/abs/2603.14602 · HTML: https://mlsanity.com/paper/2603.14602/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 24 of its 47 references that are in the map.
- First posted on arXiv 15 Mar 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 24 of 24)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Group Sequence Policy Optimization](https://mlsanity.com/paper/2507.18071/) (2025)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [ToolRL: Reward is All Tool Learning Needs](https://mlsanity.com/paper/2504.13958/) (2025)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [Compressed Chain of Thought: Efficient Reasoning Through Dense Representations](https://mlsanity.com/paper/2412.13171/) (2024)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning](https://mlsanity.com/paper/2502.03275/) (2025)
- [Moral Alignment for LLM Agents](https://mlsanity.com/paper/2410.01639/) (2024)
- [Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model](https://mlsanity.com/paper/2502.08820/) (2025)
- [Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoning](https://mlsanity.com/paper/2509.23292/) (2025)
- [DAGGER: Distractor-Aware Graph Generation for Executable Reasoning in Math Problems](https://mlsanity.com/paper/2601.06853/) (2026)
- [ActionStudio: A Lightweight Framework for Data and Training of Large Action Models](https://mlsanity.com/paper/2503.22673/) (2025)
- [GanitLLM: Difficulty-Aware Bengali Mathematical Reasoning through Curriculum-GRPO](https://mlsanity.com/paper/2601.06767/) (2026)
- [Confucius3-Math: A Lightweight High-Performance Reasoning LLM for Chinese K-12 Mathematics Learning](https://mlsanity.com/paper/2506.18330/) (2025)
- [Veri-R1: Toward Precise and Faithful Claim Verification via Online Reinforcement Learning](https://mlsanity.com/paper/2510.01932/) (2025)

## Built on by (most cited first, 4 of 4)

- [Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation](https://mlsanity.com/paper/2605.09253/) (2026)
- [DecomposeRL: Learning to Ask Useful, Informative, and Diverse Questions for Semi-Supervised, Traceable Claim Verification](https://mlsanity.com/paper/2605.27858/) (2026)
- [OracleZoom: On-Policy Self-Distillation Inspired Reference-Constrained Recursive Image Super Resolution](https://mlsanity.com/paper/2609.06490/) (2026)
- [BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering](https://mlsanity.com/paper/2609.06922/) (2026)

## Abstract (arXiv)

Conversational assistants powered by large language models (LLMs) excel at tool-use tasks but struggle with adhering to complex, business-specific rules. While models can reason over business rules provided in context, including all policies for every query introduces high latency and wastes compute. Furthermore, these lengthy prompts lead to long contexts, harming overall performance due to the 'needle-in-a-haystack' problem. To address these challenges, we propose a multi-stage alignment method that teaches models to recall and apply relevant business policies during chain-of-thought reasoning at inference time, without including the full business policy in-context. Furthermore, we introduce a novel Policy Recall reward based on the Jaccard score and a Hallucination Penalty for GRPO training. Altogether, our best model outperforms the baseline by 16 points and surpasses comparable in-context baselines of similar model size by 3 points, while using 40% fewer words.