# Novel Policy Seeking with Constrained Optimization

arXiv: https://arxiv.org/abs/2005.10696 · HTML: https://mlsanity.com/paper/2005.10696/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2020.

## Built on by (most cited first, 11 of 11)

- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization](https://mlsanity.com/paper/2204.02246/) (2022)
- [Discovering Diverse Nearly Optimal Policies with Successor Features](https://mlsanity.com/paper/2106.00669/) (2021)
- [Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond](https://mlsanity.com/paper/2310.06147/) (2023)
- [Discovering Diverse Athletic Jumping Strategies](https://mlsanity.com/paper/2105.00371/) (2021)
- [Diversifying AI: Towards Creative Chess with AlphaZero](https://mlsanity.com/paper/2308.09175/) (2023)
- [Efficient Reinforcement Learning for Global Decision Making in the Presence of Local Agents at Scale](https://mlsanity.com/paper/2403.00222/) (2024)
- [Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control](https://mlsanity.com/paper/2310.11138/) (2023)
- [Iteratively Learn Diverse Strategies with State Distance Information](https://mlsanity.com/paper/2310.14509/) (2023)
- [β-DQN: Improving Deep Q-Learning By Evolving the Behavior](https://mlsanity.com/paper/2501.00913/) (2025)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)

## Abstract (arXiv)

In problem-solving, we humans can come up with multiple novel solutions to the same problem. However, reinforcement learning algorithms can only produce a set of monotonous policies that maximize the cumulative reward but lack diversity and novelty. In this work, we address the problem of generating novel policies in reinforcement learning tasks. Instead of following the multi-objective framework used in existing methods, we propose to rethink the problem under a novel perspective of constrained optimization. We first introduce a new metric to evaluate the difference between policies and then design two practical novel policy generation methods following the new perspective. The two proposed methods, namely the Constrained Task Novel Bisector (CTNB) and the Interior Policy Differentiation (IPD), are derived from the feasible direction method and the interior point method commonly known in the constrained optimization literature. Experimental comparisons on the MuJoCo control suite show our methods can achieve substantial improvement over previous novelty-seeking methods in terms of both the novelty of policies and their performances in the primal task.