mlsanityData to 14 Oct 2026

Harness Policy Distillation: Aligning Agent Policies with Runtime Semantics

arXiv 2610.20261 ↗First posted 14 Oct 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Deploying a general-purpose model as an agent means running it inside a product-specific runtime harness that decides which actions are executable, how state advances, and when the loop may terminate. Ordinary trajectory supervision shows which action was taken but leaves the runtime semantics behind that action implicit. We introduce Harness Policy Distillation (HPD), which makes these runtime control semantics an explicit training target alongside actions to align the harness policy. HPD replays reference trajectories through the harness to recover the runtime semantics that are implicit in ordinary trajectory data, and represents them as control documents at each model-call boundary. These documents provide privileged information for a teacher to generate actions, while harness verification ensures that the resulting targets are executable under the runtime contract. We jointly train a shared model on the runtime semantics recovered from control-state privileged information and native history, thereby internalizing these semantics into the model policy. At deployment, the trained model uses the original harness interface without requiring the control document or any additional supplement. On TRAJECT-Bench with the Pi harness, Qwen3-8B trained with HPD reaches 22.98% autonomous task success and 49.23% runtime compliance, against 3.09% and 19.90% for the unadapted model. Harness-verified teacher actions substantially improve the native deployment policy, while joint control-state supervision provides further gains in task completion and runtime compliance. HPD turns executable runtime semantics into dense supervision for learning harness-compatible actions at each decision boundary.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 43 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

37 of 43 references.

References · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  12. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
25 more
  1. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  5. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  6. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  7. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  8. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  11. 2025Group Sequence Policy Optimizationcited by 835
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  13. 2016Sequence-Level Knowledge Distillationcited by 801
  14. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  15. 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
  16. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 455
  17. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  18. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  19. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 248
  20. 2015Unifying distillation and privileged informationcited by 233
  21. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
  22. 2026Privileged Information Distillation for Language Modelscited by 90
  23. 2025TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Usecited by 51
  24. 2026TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Trainingcited by 17
  25. 2026Training with Harnesses: On-Policy Harness Self-Distillation for Complex Reasoningcited by 10

Built on by

Not cited yet.