# Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training

arXiv: https://arxiv.org/abs/2604.01499 · HTML: https://mlsanity.com/paper/2604.01499/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 12 of its 20 references that are in the map.
- First posted on arXiv 2 Apr 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Builds on (most cited first, 12 of 12)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [RL's Razor: Why Online Reinforcement Learning Forgets Less](https://mlsanity.com/paper/2509.04259/) (2025)
- [Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting](https://mlsanity.com/paper/2510.18874/) (2025)
- [Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning](https://mlsanity.com/paper/2509.24372/) (2025)
- [Evolution Strategies at the Hyperscale](https://mlsanity.com/paper/2511.16652/) (2025)
- [ES Is More Than Just a Traditional Finite-Difference Approximator](https://mlsanity.com/paper/1712.06568/) (2017)
- [PCA of high dimensional random walks with comparison to neural network training](https://mlsanity.com/paper/1806.08805/) (2018)
- [The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective](https://mlsanity.com/paper/2602.00170/) (2026)
- [Evolutionary Strategies lead to Catastrophic Forgetting in LLMs](https://mlsanity.com/paper/2601.20861/) (2026)
- [ESSA: Evolutionary Strategies for Scalable Alignment](https://mlsanity.com/paper/2507.04453/) (2025)
- [High-performance Evolutionary Algorithms for Online Neuron Control](https://mlsanity.com/paper/2204.06765/) (2022)

## Built on by (most cited first, 8 of 8)

- [Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies](https://mlsanity.com/paper/2605.30148/) (2026)
- [Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements](https://mlsanity.com/paper/2608.17310/) (2026)
- [Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies](https://mlsanity.com/paper/2608.12679/) (2026)
- [Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging](https://mlsanity.com/paper/2608.05541/) (2026)
- [Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO](https://mlsanity.com/paper/2608.27351/) (2026)
- [Knowing in Advance When an Evolutionary Outer Loop Will Not Help: A Pre-Registered Cheap-Baseline Screening Rule](https://mlsanity.com/paper/2606.29119/) (2026)
- [EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale](https://mlsanity.com/paper/2609.10980/) (2026)
- [Asynchronous Is Nearly Free for Evolution Strategies on Long-Horizon Agentic Tasks](https://mlsanity.com/paper/2610.04196/) (2026)

## Abstract (arXiv)

Evolution Strategies (ES) have emerged as a scalable gradient-free alternative to reinforcement learning based LLM fine-tuning, but it remains unclear whether comparable task performance implies comparable solutions in parameter space. We compare ES and Group Relative Policy Optimization (GRPO) across four tasks in both single-task and sequential continual-learning settings. ES matches or exceeds GRPO in single-task accuracy and remains competitive sequentially when its iteration budget is controlled. Despite this similarity in task performance, the two methods produce markedly different model updates: ES makes much larger changes and induces broader off-task KL drift, whereas GRPO makes smaller, more localized updates. Strikingly, the ES and GRPO solutions are linearly connected with no loss barrier, even though their update directions are nearly orthogonal. We develop an analytical theory of ES that explains all these phenomena within a unified framework, showing how ES can accumulate large off-task movement on weakly informative directions while still making enough progress on the task to match gradient-based RL in downstream accuracy. These results show that gradient-free and gradient-based fine-tuning can reach similarly accurate yet geometrically distinct solutions, with important consequences for forgetting and knowledge preservation. The source code is publicly available: https://github.com/Bhoy1/ESvsGRPO.