mlsanityData to 14 Oct 2026

Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements

arXiv 2608.17310 ↗First posted 18 Aug 2026cs.LGcited by 3

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Reinforcement Learning (RL) has been promising in single-turn LLM fine-tuning. However, long-horizon agentic reasoning introduces increasingly branching interactions and sparse rewards, exposing several limitations of RL: its heavyweight backpropagation-based training stack makes it impractical to fine-tune larger LLMs, and longer-horizon trajectories make credit assignment in RL substantially harder. This paper argues that evolution strategies (ES) can be a better choice for fine-tuning long-horizon LLM agents. Compared with agentic RL, ES offers three key advantages: 1) Model Scalability: ES enables full-parameter optimization with only minimal, inference-level GPU memory, making it possible to fine-tune large LLMs. 2) Flexibility: its lightweight, black-box feedback interface makes ES fine-tuning easy to compose with prompt-space evolution (e.g., skill optimization & test-time compute); and 3) Long-Horizon Scalability: ES performs trajectory-level parameter attribution without decomposing rewards across horizons, yielding better scalability than Agentic RL as the horizon length grows. Based on this insight, we propose Agentic ESOpt, a full-parameter agentic fine-tuning framework tailored to flexible parameter--context co-evolution. At each step, Agentic ESOpt samples perturbations around the current LLM parameters, evaluates the resulting agents with rewards, and applies an online reward-weighted update. To improve the exploration--adaptation trade-off, Agentic ESOpt further introduces a cosine decay schedule of the perturbation scale σ. On WebArena-Lite, full-parameter optimization of Qwen-3.5-27B improves the No Skill baseline by 6.69%. In test-time automatic heuristic design, Agentic ESOpt performs online prompt--parameter co-evolution, improving its matched baseline in 28 of 36 settings.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 41 of its 56 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

41 of 56 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning8 Jul
  2. 2026Milestone-Guided Policy Learning for Long-Horizon Language Agents7 May
  3. 2026From Reasoning to Agentic: Credit Assignment in Reinforcement Learning for Large Language Models10 Apr
  4. 2025GEM: A Gym for Agentic LLMs1 Oct
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May

Other references · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  11. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  12. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
24 more
  1. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  3. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 574
  4. 2024Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Modelcited by 268
  5. 2026Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skillscited by 187
  6. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 163
  7. 2026SkillOpt: Executive Strategy for Self-Evolving Agent Skillscited by 148
  8. 2026Kimi K3: Open Frontier Intelligencecited by 138
  9. 2025From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMscited by 124
  10. 2025Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Designcited by 109
  11. 2025Scaling Test-time Compute for LLM Agentscited by 63
  12. 2026Maximum Likelihood Reinforcement Learningcited by 36
  13. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 28
  14. 2025Evolution Strategies at the Hyperscalecited by 25
  15. 2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimizationcited by 21
  16. 2026SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibecited by 20
  17. 2026Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Trainingcited by 8
  18. 2025ESSA: Evolutionary Strategies for Scalable Alignmentcited by 6
  19. 2025Fine-tuning Large Language Model for Automated Algorithm Designcited by 6
  20. 2026On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Lengthcited by 6
  21. 2025EA4LLM: A Gradient-Free Approach to Large Language Model Optimization via Evolutionary Algorithmscited by 5
  22. 2026ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuningcited by 5
  23. 2026Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Modelscited by 2
  24. 2026Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Mergingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPOcited by 1
  2. 2026AIM-ZO: Activation-Informed Subspace Maintenance for Zeroth-Order LLM Fine-Tuningcited by 0
  3. 2026Asynchronous Is Nearly Free for Evolution Strategies on Long-Horizon Agentic Taskscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
  2. 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agents
  3. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning
  4. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
  5. 2026EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learning
  6. 2026Beyond the Capability Boundary: Zeroth-Order Optimization for Self-Evolving LLM Agents
  7. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
  8. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution