# Making Reinforcement Learning Work on Swimmer

arXiv: https://arxiv.org/abs/2208.07587 · HTML: https://mlsanity.com/paper/2208.07587/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 21 references that are in the map.
- First posted on arXiv 16 Aug 2022.

## Builds on (most cited first, 11 of 11)

- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Training Agents using Upside-Down Reinforcement Learning](https://mlsanity.com/paper/1912.02877/) (2019)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)
- [Evolution-Guided Policy Gradient in Reinforcement Learning](https://mlsanity.com/paper/1805.07917/) (2018)
- [CEM-RL: Combining evolutionary and gradient-based methods for policy search](https://mlsanity.com/paper/1810.01222/) (2018)
- [Heuristic-Guided Reinforcement Learning](https://mlsanity.com/paper/2106.02757/) (2021)
- [Proximal Distilled Evolutionary Reinforcement Learning](https://mlsanity.com/paper/1906.09807/) (2019)
- [A Surrogate-Assisted Controller for Expensive Evolutionary Reinforcement Learning](https://mlsanity.com/paper/2201.00129/) (2022)
- [Genetic-Gated Networks for Deep Reinforcement](https://mlsanity.com/paper/1903.01886/) (2018)
- [Cooperative Heterogeneous Deep Reinforcement Learning](https://mlsanity.com/paper/2011.00791/) (2020)
- [FiDi-RL: Incorporating Deep Reinforcement Learning with Finite-Difference Policy Search for Efficient Learning of Continuous Control](https://mlsanity.com/paper/1907.00526/) (2019)

## Built on by (most cited first, 6 of 6)

- [SINDy-RL: Interpretable and Efficient Model-Based Reinforcement Learning](https://mlsanity.com/paper/2403.09110/) (2024)
- [Seizing Serendipity: Exploiting the Value of Past Success in Off-Policy Actor-Critic](https://mlsanity.com/paper/2306.02865/) (2023)
- [Average-Reward Soft Actor-Critic](https://mlsanity.com/paper/2501.09080/) (2025)
- [Pontryagin Optimal Control via Neural Networks](https://mlsanity.com/paper/2212.14566/) (2022)
- [Evolutionary Strategy Guided Reinforcement Learning via MultiBuffer Communication](https://mlsanity.com/paper/2306.11535/) (2023)
- [SP3O: Reinforcement Learning from Segment Preferences without Reward Modeling](https://mlsanity.com/paper/2608.02951/) (2026)

## Abstract (arXiv)

The SWIMMER environment is a standard benchmark in reinforcement learning (RL). In particular, it is often used in papers comparing or combining RL methods with direct policy search methods such as genetic algorithms or evolution strategies. A lot of these papers report poor performance on SWIMMER from RL methods and much better performance from direct policy search methods. In this technical report we show that the low performance of RL methods on SWIMMER simply comes from the inadequate tuning of an important hyper-parameter, the discount factor. Furthermore we show that, by setting this hyper-parameter to a correct value, the issue can be easily fixed. Finally, for a set of often used RL algorithms, we provide a set of successful hyper-parameters obtained with the Stable Baselines3 library and its RL Zoo.