mlsanityData to 7 Oct 2026

Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning

arXiv 2509.24372 ↗First posted 29 Sep 2025cs.LG cs.AI cs.NEcited by 28

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 60 of its 90 references that are in the map. First posted on arXiv 29 Sep 2025.

Builds on

60 of 90 references.

References · 60

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
38 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  3. 2024Mixtral of Expertscited by 1,785
  4. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  6. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2023BloombergGPT: A Large Language Model for Financecited by 923
  10. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  11. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  13. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  14. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
  15. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  16. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 402
  17. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  18. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  19. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  20. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  21. 2024Evolutionary Optimization of Model Merging Recipescited by 204
  22. 2017Parameter Space Noise for Explorationcited by 158
  23. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 158
  24. 2025Qwen2.5-1M Technical Reportcited by 158
  25. 2017Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learningcited by 148
  26. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  27. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  28. 2023Language Model Crossover: Variation through Few-Shot Promptingcited by 119
  29. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  30. 2024ARC Prize 2024: Technical Reportcited by 103
  31. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  32. 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 89
  33. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  34. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  35. 2022BBTv2: Towards a Gradient-Free Future with Large Language Modelscited by 53
  36. 2024Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuningcited by 53
  37. 2023Fine-tuning ChatGPT for Automatic Scoringcited by 50
  38. 2018Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Ataricited by 26

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 24.

28 papers.

Sort:

By area:

  1. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 25
  2. 2025Evolution Strategies at the Hyperscalecited by 22
  3. 2026The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspectivecited by 9
  4. 2026Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Trainingcited by 7
  5. 2026Evolutionary Strategies lead to Catastrophic Forgetting in LLMscited by 6
  6. 2025EA4LLM: A Gradient-Free Approach to Large Language Model Optimization via Evolutionary Algorithmscited by 5
  7. 2026Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Costcited by 5
  8. 2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 3
  9. 2026Hard Constraints Meet Soft Generation: Guaranteed Feasibility for LLM-based Combinatorial Optimizationcited by 3
  10. 2025Mistake Notebook Learning: Batch-Clustered Failures for Training-Free Agent Adaptationcited by 2
  11. 2026Fine-Tuning Language Models to Know What They Knowcited by 2
  12. 2026PopuLoRA: Co-Evolving LLM Populations for Reasoning Self-Playcited by 2
16 more
  1. 2026Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategiescited by 2
  2. 2026Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategiescited by 2
  3. 2026Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundarycited by 1
  4. 2026Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Mergingcited by 1
  5. 2026Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPOcited by 1
  6. 2026Coverage, Not Credit: Failure-Credit Routing of Zeroth-Order Perturbation Budgets Does Not Improve On-Pool Sample Efficiency for LLM Agentscited by 1
  7. 2026Optimize Wider, Not Deeper: Consensus Aggregation for Policy Optimizationcited by 0
  8. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuningcited by 0
  9. 2026Inference-Native Zeroth-Order Optimization for LLMscited by 0
  10. 2026Genetic Algorithms with Optimization Guided Operatorscited by 0
  11. 2026Reward-Aware Population Scaling of Evolutionary Strategies in LLM Fine-Tuningcited by 0
  12. 2026Can Bayesian Optimization Efficiently Find a Strong Single Expert in Neural Thickets?cited by 0
  13. 2026Hill Sampling for Test-Time Scaling: A Simple and Better Alternative to Repeated Sampling, Evolution, and Trainingcited by 0
  14. 2026Modular Norm RandOpt: Population-Efficient Ensembling through Architecture-Aware Perturbationscited by 0
  15. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edgecited by 0
  16. 2026Continual Reinforcement Learning with Neuroevolutioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements
  2. 2026Position: Agentic Evolution is the Path to Evolving LLMs
  3. 2026ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning
  4. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
  5. 2025A Survey on the Optimization of Large Language Model-based Agents
  6. 2025On Predictability of Reinforcement Learning Dynamics for Large Language Models
  7. 2024Self-Evolution Fine-Tuning for Policy Optimization
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs