# Learning Novel Policies For Tasks

arXiv: https://arxiv.org/abs/1905.05252 · HTML: https://mlsanity.com/paper/1905.05252/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 May 2019.

## Built on by (most cited first, 20 of 20)

- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain](https://mlsanity.com/paper/2109.06668/) (2021)
- [Diversity-Aware Policy Optimization for Large Language Model Reasoning](https://mlsanity.com/paper/2505.23433/) (2025)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [PolyNet: Learning Diverse Solution Strategies for Neural Combinatorial Optimization](https://mlsanity.com/paper/2402.14048/) (2024)
- [Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery](https://mlsanity.com/paper/2210.03516/) (2022)
- [Discovering Diverse Nearly Optimal Policies with Successor Features](https://mlsanity.com/paper/2106.00669/) (2021)
- [Discovering Diverse Athletic Jumping Strategies](https://mlsanity.com/paper/2105.00371/) (2021)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [Iteratively Learn Diverse Strategies with State Distance Information](https://mlsanity.com/paper/2310.14509/) (2023)
- [Learning Diverse Policies with Soft Self-Generated Guidance](https://mlsanity.com/paper/2402.04539/) (2024)
- [Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization](https://mlsanity.com/paper/2511.01374/) (2025)
- [Influence-Based Reinforcement Learning for Intrinsically-Motivated Agents](https://mlsanity.com/paper/2108.12581/) (2021)
- [Adaptive trajectory-constrained exploration strategy for deep reinforcement learning](https://mlsanity.com/paper/2312.16456/) (2023)
- [Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptation](https://mlsanity.com/paper/2209.12029/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Preventing Imitation Learning with Adversarial Policy Ensembles](https://mlsanity.com/paper/2002.01059/) (2020)
- [Policy Dispersion in Non-Markovian Environment](https://mlsanity.com/paper/2302.14509/) (2023)
- [Diversity for Contingency: Learning Diverse Behaviors for Efficient Adaptation and Transfer](https://mlsanity.com/paper/2310.07493/) (2023)
- [Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agents](https://mlsanity.com/paper/2609.38805/) (2026)

## Abstract (arXiv)

In this work, we present a reinforcement learning algorithm that can find a variety of policies (novel policies) for a task that is given by a task reward function. Our method does this by creating a second reward function that recognizes previously seen state sequences and rewards those by novelty, which is measured using autoencoders that have been trained on state sequences from previously discovered policies. We present a two-objective update technique for policy gradient algorithms in which each update of the policy is a compromise between improving the task reward and improving the novelty reward. Using this method, we end up with a collection of policies that solves a given task as well as carrying out action sequences that are distinct from one another. We demonstrate this method on maze navigation tasks, a reaching task for a simulated robot arm, and a locomotion task for a hopper. We also demonstrate the effectiveness of our approach on deceptive tasks in which policy gradient methods often get stuck.