# Bilevel Optimization of Agent Skills via Monte Carlo Tree Search

arXiv: https://arxiv.org/abs/2604.15709 · HTML: https://mlsanity.com/paper/2604.15709/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Agent skill learning and evolution, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 23 references that are in the map.
- First posted on arXiv 17 Apr 2026.

## Placed in

- LLM agents and tool use / [Agent skill learning and evolution](https://mlsanity.com/agents/agent-skill-learning-and-evolution/)

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Reasoning with Language Model is Planning with World Model](https://mlsanity.com/paper/2305.14992/) (2023)
- [Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models](https://mlsanity.com/paper/2310.04406/) (2023)
- [AFlow: Automating Agentic Workflow Generation](https://mlsanity.com/paper/2410.10762/) (2024)
- [SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks](https://mlsanity.com/paper/2602.12670/) (2026)
- [rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking](https://mlsanity.com/paper/2501.04519/) (2025)
- [Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8B](https://mlsanity.com/paper/2406.07394/) (2024)
- [Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale](https://mlsanity.com/paper/2601.10338/) (2026)
- [Reinforcement Learning for Self-Improving Agent with Skill Library](https://mlsanity.com/paper/2512.17102/) (2025)
- [Organizing, Orchestrating, and Benchmarking Agent Skills at Ecosystem Scale](https://mlsanity.com/paper/2603.02176/) (2026)
- [Evaluating LLM Reasoning in the Operations Research Domain with ORQA](https://mlsanity.com/paper/2412.17874/) (2024)
- [Surrogate-Based Simulation Optimization](https://mlsanity.com/paper/2105.03893/) (2021)
- [A Contextual Ranking and Selection Method for Personalized Medicine](https://mlsanity.com/paper/2206.12640/) (2022)

## Built on by (most cited first, 4 of 4)

- [Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries](https://mlsanity.com/paper/2607.10113/) (2026)
- [Learning to Learn from Multimodal Experience](https://mlsanity.com/paper/2605.16857/) (2026)
- [Safe Bilevel Delegation (SBD): A Formal Framework for Runtime Delegation Safety in Multi-Agent Systems](https://mlsanity.com/paper/2604.27358/) (2026)
- [AtomSearch: Evidence-Grounded Tree Search for Agent Skill Optimization](https://mlsanity.com/paper/2610.19951/) (2026)

## Abstract (arXiv)

Agent textttskills are structured collections of instructions, tools, and supporting resources that help large language model (LLM) agents perform particular classes of tasks. Empirical evidence shows that the design of textttskills can materially affect agent task performance, yet systematically optimizing textttskills remains challenging. Since a textttskill comprises instructions, tools, and supporting resources in a structured way, optimizing it requires jointly determining both the structure of these components and the content each component contains. This gives rise to a complex decision space with strong interdependence across structure and components. We therefore represent these two coupled decisions as textttskill structure and component content, and formulate textttskill optimization as a bilevel optimization problem. We propose a bilevel optimization framework in which an outer loop employs Monte Carlo Tree Search to determine the textttskill structure, while an inner loop refines the component content within the structure selected by the outer loop. In both loops, we employ LLMs to assist the optimization procedure. We evaluate the proposed framework on an open-source Operations Research Question Answering dataset, and the experimental results suggest that the bilevel optimization framework improves the performance of the agents with the optimized textttskill.