# GraphAllocBench: A Flexible Benchmark for Preference-Conditioned Multi-Objective Policy Learning

arXiv: https://arxiv.org/abs/2601.20753 · HTML: https://mlsanity.com/paper/2601.20753/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 25 of its 52 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2410.21845/) (2024)
- [A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation](https://mlsanity.com/paper/1908.08342/) (2019)
- [A Survey of Deep Reinforcement Learning in Video Games](https://mlsanity.com/paper/1912.10944/) (2019)
- [Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2112.11701/) (2021)
- [PD-MORL: Preference-Driven Multi-Objective Reinforcement Learning Algorithm](https://mlsanity.com/paper/2208.07914/) (2022)
- [Pareto Set Learning for Neural Multi-objective Combinatorial Optimization](https://mlsanity.com/paper/2203.15386/) (2022)
- [Smooth Tchebycheff Scalarization for Multi-Objective Optimization](https://mlsanity.com/paper/2402.19078/) (2024)
- [Scaling Pareto-Efficient Decision Making Via Offline Multi-Objective RL](https://mlsanity.com/paper/2305.00567/) (2023)
- [Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity](https://mlsanity.com/paper/2110.04041/) (2021)
- [Pareto Set Learning for Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2501.06773/) (2025)
- [Reinforcement Learning Within the Classical Robotics Stack: A Case Study in Robot Soccer](https://mlsanity.com/paper/2412.09417/) (2024)
- [Double Oracle Algorithm for Game-Theoretic Robot Allocation on Graphs](https://mlsanity.com/paper/2312.11791/) (2023)
- [Federated reinforcement learning for robot motion planning with zero-shot generalization](https://mlsanity.com/paper/2403.13245/) (2024)
- [Role Play: Learning Adaptive Role-Specific Strategies in Multi-Agent Interactions](https://mlsanity.com/paper/2411.01166/) (2024)
- [Dynamic Defender-Attacker Blotto Game](https://mlsanity.com/paper/2112.09890/) (2021)
- [A Novel Pareto-optimal Ranking Method for Comparing Multi-objective Optimization Algorithms](https://mlsanity.com/paper/2411.17999/) (2024)
- [GrapheonRL: A Graph Neural Network and Reinforcement Learning Framework for Constraint and Data-Aware Workflow Mapping and Scheduling in Heterogeneous HPC Systems](https://mlsanity.com/paper/2506.00260/) (2025)
- [A Review of the Deep Sea Treasure problem as a Multi-Objective Reinforcement Learning Benchmark](https://mlsanity.com/paper/2110.06742/) (2021)
- [Reinforcement Learning for Game-Theoretic Resource Allocation on Graphs](https://mlsanity.com/paper/2505.06319/) (2025)
- [RLHGNN: Reinforcement Learning-driven Heterogeneous Graph Neural Network for Next Activity Prediction in Business Processes](https://mlsanity.com/paper/2507.02690/) (2025)

## Built on by (most cited first, 2 of 2)

- [Controllability in preference-conditioned multi-objective reinforcement learning](https://mlsanity.com/paper/2605.10585/) (2026)
- [PROMO: Preference-conditioned Multi-Objective Reinforcement Learning for Quadrupedal Robots](https://mlsanity.com/paper/2610.01260/) (2026)

## Abstract (arXiv)

Preference-Conditioned Policy Learning (PCPL) in Multi-Objective Reinforcement Learning (MORL) approximates diverse Pareto-optimal solutions by conditioning a single policy on user-specified preferences, enabling run-time adaptation to arbitrary trade-offs without retraining. However, existing PCPL benchmarks are largely restricted to toy tasks and fixed environments, limiting their realism and scalability. To address this gap, we introduce GraphAllocBench, a flexible benchmark built on CityPlannerEnv, a novel graph-based resource allocation sandbox inspired by city management. GraphAllocBench provides a rich suite of problems with customizable objective functions, varying preference conditions, complex Pareto Fronts, and high-dimensional scalability. We further propose two supplementary metrics -- Proportion of Non-Dominated Solutions (PNDS) and Ordering Score (OS) -- that capture prediction reliability and preference consistency while complementing the widely used hypervolume metric. Through experiments with several state-of-the-art PCPL algorithms and our own MLP and graph-aware PCPL-PPO baseline, we show that GraphAllocBench exposes distinct failure modes that hypervolume alone does not capture but our supplementary metrics reveal, while motivating graph-based approaches such as Graph Neural Networks (GNNs) for scaling to complex, high-dimensional allocation tasks. By letting users freely vary objectives, preferences, and allocation rules, GraphAllocBench serves as a versatile and extensible testbed for advancing PCPL.