Offline Reinforcement Learning of High-Quality Behaviors Under Robust Style Alignment
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
We study offline reinforcement learning of style-conditioned policies using explicit style supervision via subtrajectory labeling functions. In this setting, aligning style with high task performance is particularly challenging due to distribution shift and inherent conflicts between style and reward. Existing methods, despite introducing numerous definitions of style, often fail to reconcile these objectives effectively. To address these challenges, we propose a unified definition of behavior style and instantiate it into a practical framework. Building on this, we introduce Style-Conditioned Implicit Q-Learning (SCIQL), which leverages offline goal-conditioned RL techniques, such as hindsight relabeling and value learning, and combine it with a new Gated Advantage Weighted Regression mechanism to efficiently optimize task performance while preserving style alignment. Experiments demonstrate that SCIQL achieves superior performance on both objectives compared to prior offline methods. Code, datasets and visuals are available in: https://mathieu-petitbois.github.io/projects/sciql/.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 67 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
45 of 67 references.
Closest ancestors · 8
Same direction, newest first
- 2024AD4RL: Autonomous Driving Benchmarks for Offline Reinforcement Learning with Value-based Dataset3 Apr
- 2024Policy-regularized Offline Multi-objective Reinforcement Learning4 Jan
- 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
Other references · 37
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
- 2015Trust Region Policy Optimizationcited by 2,503
- 2016Generative Adversarial Imitation Learningcited by 1,576
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
- 2017Hindsight Experience Replaycited by 1,096
- 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
- 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
- 2021Implicit Behavioral Cloningcited by 442
- 2022Behavior Transformers: Cloning k modes with one stonecited by 299
25 more
- 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
- 2023Imitating Human Behaviour with Diffusion Modelscited by 260
- 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
- 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
- 2024Behavior Generation with Latent Actionscited by 159
- 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
- 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
- 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
- 2021Learning One Representation to Optimize All Rewardscited by 110
- 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
- 2017Robust Imitation of Diverse Behaviorscited by 85
- 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
- 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrationscited by 51
- 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
- 2019Learning to Reach Goals via Iterated Supervised Learningcited by 26
- 2019Learning Calibratable Policies using Programmatic Style-Consistencycited by 11
- 2024An Offline Adaptation Framework for Constrained Multi-Objective Reinforcement Learningcited by 5
- 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planningcited by 5
- 2024MODULI: Unlocking Preference Generalization via Diffusion Models for Offline Multi-Objective Reinforcement Learningcited by 4
- 2024Diverse Policies Recovering via Pointwise Mutual Information Weighted Imitation Learningcited by 4
- 2021An Unsupervised Video Game Playstyle Metric via State Discretizationcited by 3
- 2024Perceptual Similarity for Measuring Decision-Making Style and Policy Diversity in Gamescited by 2
- 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
- 2025Offline Learning of Controllable Diverse Behaviorscited by 2
- 2025Constrained Style Learning from Imperfect Demonstrations under Task Optimalitycited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
- 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning
- 2026Causal Flow Q-Learning for Robust Offline Reinforcement Learning
- 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
- 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
- 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
- 2025Flow Q-Learning