# ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning

arXiv: https://arxiv.org/abs/2602.01003 · HTML: https://mlsanity.com/paper/2602.01003/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- First posted on arXiv 1 Feb 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)

## Built on by (most cited first, 5 of 5)

- [Agentic ESOpt: Fine-Tuning Long-Horizon LLM Agents with Minimal GPU Requirements](https://mlsanity.com/paper/2608.17310/) (2026)
- [Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training](https://mlsanity.com/paper/2608.02391/) (2026)
- [Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging](https://mlsanity.com/paper/2608.05541/) (2026)
- [Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO](https://mlsanity.com/paper/2608.27351/) (2026)
- [DART-ES: Difficulty-Aware Reweighting and Targeted Replay for Fine-Tuning LLMs with Evolution Strategies](https://mlsanity.com/paper/2610.06993/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) has become a key training step for improving mathematical reasoning in large language models (LLMs), but it often has high GPU memory usage, which makes it hard to use in settings with limited resources. To reduce these issues, we propose Evolution Strategies with Sharpness-Aware Maximization (ESSAM), a full parameter fine-tuning framework that tightly combines the zero-order search in parameter space from Evolution Strategies (ES) with the Sharpness-Aware Maximization (SAM) to improve generalization. We conduct fine-tuning experiments on the mainstream mathematica reasoning task GSM8K. The results show that ESSAM achieves an average accuracy of 78.27% across all models and its overall performance is comparable to RL methods. It surpasses classic RL algorithm PPO with an accuracy of 77.72% and is comparable to GRPO with an accuracy of 78.34%, and even surpassing them on some models. Further generalization experiments show that the models trained with ESSAM exhibit stronger generalization ability. Their average performance achieves the best results on 5 out of 6 datasets, indicating that ESSAM can effectively improve the generalization performance of fine-tuned models. In terms of GPU memory usage, ESSAM reduces the average GPU memory usage by 18× compared to PPO and by 10× compared to GRPO, achieving an extremely low GPU memory usage. In addition, we design an accelerated variant of ESSAM, which achieves nearly a twofold speedup while maintaining the same GPU memory usage as ESSAM, and attains an average accuracy of 78.02% across all models, outperforming PPO. Code: https://github.com/szs777/ESSAM