# Learning Control Policies for Variable Objectives from Offline Data

arXiv: https://arxiv.org/abs/2308.06127 · HTML: https://mlsanity.com/paper/2308.06127/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 23 of its 38 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 23 of 23)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [MOReL : Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2005.05951/) (2020)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research](https://mlsanity.com/paper/1802.09464/) (2018)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning](https://mlsanity.com/paper/2204.12581/) (2022)
- [Temporal Difference Models: Model-Free Deep RL for Model-Based Control](https://mlsanity.com/paper/1802.09081/) (2018)
- [NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.00714/) (2021)
- [Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks](https://mlsanity.com/paper/1605.07127/) (2016)
- [Overcoming Model Bias for Robust Offline Deep Reinforcement Learning](https://mlsanity.com/paper/2008.05533/) (2020)
- [A Benchmark Environment Motivated by Industrial Control Problems](https://mlsanity.com/paper/1709.09480/) (2017)
- [Confidence-Conditioned Value Functions for Offline Reinforcement Learning](https://mlsanity.com/paper/2212.04607/) (2022)
- [User-Interactive Offline Reinforcement Learning](https://mlsanity.com/paper/2205.10629/) (2022)
- [Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learning](https://mlsanity.com/paper/2201.05433/) (2022)
- [Batch Reinforcement Learning on the Industrial Benchmark: First Experiences](https://mlsanity.com/paper/1705.07262/) (2017)

## Built on by (most cited first, 3 of 3)

- [Why long model-based rollouts are no reason for bad Q-value estimates](https://mlsanity.com/paper/2407.11751/) (2024)
- [Reward Dimension Reduction for Scalable Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2502.20957/) (2025)
- [Constrained Reinforcement Learning Using Successor Representations](https://mlsanity.com/paper/2607.24057/) (2026)

## Abstract (arXiv)

Offline reinforcement learning provides a viable approach to obtain advanced control strategies for dynamical systems, in particular when direct interaction with the environment is not available. In this paper, we introduce a conceptual extension for model-based policy search methods, called variable objective policy (VOP). With this approach, policies are trained to generalize efficiently over a variety of objectives, which parameterize the reward function. We demonstrate that by altering the objectives passed as input to the policy, users gain the freedom to adjust its behavior or re-balance optimization targets at runtime, without need for collecting additional observation batches or re-training.