# Pareto Q-Learning with Reward Machines

arXiv: https://arxiv.org/abs/2606.19134 · HTML: https://mlsanity.com/paper/2606.19134/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Interpretable and logic-guided RL, one of 15 directions in Deep reinforcement learning methods.
- Builds on 4 of its 10 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Placed in

- Deep reinforcement learning methods / [Interpretable and logic-guided RL](https://mlsanity.com/deeprl/interpretable-and-logic-guided-rl/)

## Builds on (most cited first, 4 of 4)

- [Agent57: Outperforming the Atari Human Benchmark](https://mlsanity.com/paper/2003.13350/) (2020)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Energy-aware optimization of UAV base stations placement via decentralized multi-agent Q-learning](https://mlsanity.com/paper/2106.00845/) (2021)
- [Real Robot Challenge 2022: Learning Dexterous Manipulation from Offline Data in the Real World](https://mlsanity.com/paper/2308.07741/) (2023)

## Abstract (arXiv)

We present Pareto Q-Learning with Reward Machines (PQLRM), a multi-objective reinforcement learning algorithm for tasks whose reward structure is specified by a set of reward machines (RMs). PQLRM combines Pareto Q-Learning (PQL), which maintains sets of vector-valued Q-estimates to approximate the Pareto front, with enhancements from Q-Learning with Reward Machines (QRM), which exploits the factored automaton structure of the reward signal. This yields a multi-policy algorithm that remains sample-efficient under non-Markovian, RM-encoded rewards. Experimental trials show that PQLRM converges faster than a naive PQL baseline applied to the cross-product MDP and can synthesize Pareto-optimal policies that QRM cannot.