mlsanityData to 7 Oct 2026

Utilizing Novelty-based Evolution Strategies to Train Transformers in Reinforcement Learning

arXiv 2502.06301 ↗First posted 10 Feb 2025cs.LG cs.NEcited by 1

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

In this paper, we experiment with novelty-based variants of OpenAI-ES, the NS-ES and NSR-ES algorithms, and evaluate their effectiveness in training complex, transformer-based architectures designed for the problem of reinforcement learning, such as Decision Transformers. We also test if we can accelerate the novelty-based training of these larger models by seeding the training with a pretrained models. The experimental results were mixed. NS-ES showed progress, but it would clearly need many more iterations for it to yield interesting agents. NSR-ES, on the other hand, proved quite capable of being straightforwardly used on larger models, since its performance appears as similar between the feed-forward model and Decision Transformer, as it was for the OpenAI-ES in our previous work.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 12 of its 21 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

12 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 10

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2016OpenAI Gymcited by 1,684
  4. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  5. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  6. 2021Deep Reinforcement Learning Versus Evolution Strategies: A Comparative Surveycited by 23
  7. 2022Combining Evolution and Deep Reinforcement Learning for Policy Search: a Surveycited by 20
  8. 2019Efficacy of Modern Neuro-Evolutionary Strategies for Continuous Control Optimizationcited by 15
  9. 2018First-order and second-order variants of the gradient descent in a unified frameworkcited by 3
  10. 2025Utilizing Evolution Strategies to Train Transformers in Reinforcement Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025EA4LLM: A Gradient-Free Approach to Large Language Model Optimization via Evolutionary Algorithmscited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
  2. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
  3. 2023Novelty Detection in Reinforcement Learning with World Models
  4. 2026Novelty Adaptation Through Hybrid Large Language Model (LLM)-Symbolic Planning and LLM-guided Reinforcement Learning
  5. 2023Neuro-Symbolic World Models for Adapting to Open World Novelty
  6. 2026EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning
  7. 2023Emergent Agentic Transformer from Chain of Hindsight Experience
  8. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning