mlsanityData to 14 Oct 2026

DEPO: Dual-Efficiency Preference Optimization for LLM Agents

arXiv 2511.15392 ↗First posted 19 Nov 2025cs.CL cs.AIcited by 4

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Recent advances in large language models (LLMs) have greatly improved their reasoning and decision-making abilities when deployed as agents. Richer reasoning, however, often comes at the cost of longer chain of thought (CoT), hampering interaction efficiency in real-world scenarios. Nevertheless, there still lacks systematic definition of LLM agent efficiency, hindering targeted improvements. To this end, we introduce dual-efficiency, comprising (i) step-level efficiency, which minimizes tokens per step, and (ii) trajectory-level efficiency, which minimizes the number of steps to complete a task. Building on this definition, we propose DEPO, a dual-efficiency preference optimization method that jointly rewards succinct responses and fewer action steps. Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in performance. DEPO also generalizes to three out-of-domain math benchmarks and retains its efficiency gains when trained on only 25% of the data. Our project page is at https://opencausalab.github.io/DEPO.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 51 references that are in the map. First posted on arXiv 19 Nov 2025.

Builds on

29 of 51 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Group-in-Group Policy Optimization for LLM Agent Training16 May
  2. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  3. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  9. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  10. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  11. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  12. 2025O1-Pruner: Length-Harmonizing Fine-Tuning for O1-Like Reasoning Pruningcited by 279
12 more
  1. 2024Distilling System 2 into System 1cited by 159
  2. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 158
  3. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 119
  4. 2025Sketch-of-Thought: Efficient LLM Reasoning with Adaptive Cognitive-Inspired Sketchingcited by 117
  5. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
  6. 2025Agent-R: Training Language Model Agents to Reflect via Iterative Self-Trainingcited by 82
  7. 2024Direct Multi-Turn Preference Optimization for Language Agentscited by 64
  8. 2024Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoningcited by 63
  9. 2024Can Language Models Learn to Skip Steps?cited by 59
  10. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  11. 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimizationcited by 22
  12. 2025ARise: Towards Knowledge-Augmented Reasoning via Risk-Adaptive Searchcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Agent-Omit: Adaptive Context Omission for Efficient LLM Agentscited by 5
  2. 2026Healthcare AI GYM for Medical Agentscited by 4
  3. 2026Cost-Utility Alignment in LLM Agent Trajectories:Profiling,Attribution,Diagnosis,Adaptation,and Evaluationcited by 2
  4. 2026Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2024Enhancing Decision-Making for LLM Agents via Step-Level Q-Value Models
  4. 2025SDPO: Segment-Level Direct Preference Optimization for Social Agents
  5. 2025A Survey of Direct Preference Optimization
  6. 2025SPOGW: a Score-based Preference Optimization method via Group-Wise comparison for workflows
  7. 2025OPT-BENCH: Evaluating LLM Agent on Large-Scale Search Spaces Optimization Problems
  8. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions