mlsanityData to 7 Oct 2026

Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimization

arXiv 2606.16111 ↗First posted 15 Jun 2026cs.CLcited by 2

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Recent advances in tool-integrated language agents have significantly improved their ability to solve complex reasoning tasks. However, existing alignment methods predominantly focus on maximizing task accuracy, while overlooking auxiliary objectives such as tool-use efficiency, which are essential for practical deployment. To address this gap, we introduce ParetoPO, a two-stage multi-objective optimization framework for aligning tool-using large language models (LLMs) under competing objectives. In the first stage, ParetoPO leverages hypervolume-guided dynamic scalarization to adapt reward weights based on global Pareto frontier progress. In the second stage, it replaces scalarized learning signals with Pareto-ranking-based advantage computation, promoting nondominated trajectories through dominance-aware credit assignment. This design enables fine-grained, action-level optimization across multiple conflicting objectives. Experimental results on mathematic reasoning and multi-hop QA tasks show that ParetoPO consistently discovers policies with superior accuracy-efficiency trade-offs compared to static and heuristic baselines.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 39 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

25 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Acting Less is Reasoning More! Teaching Model to Act Efficiently21 Apr

Other references · 24

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  8. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  9. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  12. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
12 more
  1. 2025Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learningcited by 68
  2. 2024Conditional Language Policy: A General Framework for Steerable Multi-Objective Finetuningcited by 42
  3. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  4. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  5. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 22
  6. 2025Incentivizing Dual Process Thinking for Efficient Large Language Model Reasoningcited by 21
  7. 2025MO-GRPO: Mitigating Reward Hacking of Group Relative Policy Optimization on Multi-Objective Problemscited by 21
  8. 2025Pareto Multi-Objective Alignment for Language Modelscited by 20
  9. 2025Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Modelscited by 18
  10. 2025Training Language Models to Generate Quality Code with Program Analysis Feedbackcited by 11
  11. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignmentcited by 10
  12. 2025TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Frameworkcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026R2-Searcher: Calibrating Retrieval and Reasoning Boundaries for Agentic Searchcited by 0
  2. 2026MOAE: Multi-Objective Agent Evolution with Pareto-Preserving Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Agent Tool-Integrated Policy Optimization
  2. 2025Tool-Augmented Policy Optimization: Synergizing Reasoning and Adaptive Tool Use with Reinforcement Learning
  3. 2025PORTool: Importance-Aware Policy Optimization with Rewarded Tree for Multi-Tool-Integrated Reasoning
  4. 2025Agentic Reinforced Policy Optimization
  5. 2026ToolSearcher: Optimizing Tool Selection at Scale via Reinforcement Learning
  6. 2026Joint Optimization of Tool Creation and Use for Large Language Model Agents
  7. 2026AdaTIR: Adaptive Tool-Integrated Reasoning via Difficulty-Aware Policy Optimization
  8. 2025AWPO: Enhancing Tool-Use of Large Language Models through Adaptive Integration of Reasoning Rewards