mlsanityData to 14 Oct 2026

APC-RL: Exceeding Data-Driven Behavior Priors with Adaptive Policy Composition

arXiv 2601.19452 ↗First posted 27 Jan 2026cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Incorporating demonstration data into reinforcement learning (RL) can greatly accelerate learning, but existing approaches often assume demonstrations are optimal and fully aligned with the target task. In practice, demonstrations are frequently sparse, suboptimal, or misaligned, which can degrade performance when these demonstrations are integrated into RL. We propose Adaptive Policy Composition (APC), a hierarchical model that adaptively composes multiple data-driven Normalizing Flow (NF) priors. Instead of enforcing strict adherence to the priors, APC estimates each prior's applicability to the target task while leveraging them for exploration. Moreover, APC either refines useful priors, or sidesteps misaligned ones when necessary to optimize downstream reward. Across diverse benchmarks, APC accelerates learning when demonstrations are aligned, remains robust under severe misalignment, and leverages suboptimal demonstrations to bootstrap exploration while avoiding performance degradation caused by overly strict adherence to suboptimal demonstrations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 41 of its 53 references that are in the map. First posted on arXiv 27 Jan 2026.

Builds on

41 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Foundation Policies with Hilbert Representations23 Feb

Other references · 40

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Variational Inference with Normalizing Flowscited by 2,170
  3. 2016Density estimation using Real NVPcited by 2,042
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2016Generative Adversarial Imitation Learningcited by 1,597
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,323
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  11. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  12. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
28 more
  1. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  2. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  3. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  4. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  5. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 171
  6. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  7. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  8. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  9. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  10. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  11. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  12. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
  13. 2023Imitation Bootstrapped Reinforcement Learningcited by 64
  14. 2017Multi-Level Discovery of Deep Optionscited by 51
  15. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  16. 2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
  17. 2019Efficient Bimanual Manipulation Using Learned Task Schemascited by 45
  18. 2019Improving Exploration in Soft-Actor-Critic with Normalizing Flows Policiescited by 38
  19. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  20. 2022Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Roboticscited by 24
  21. 2020Deep Inverse Q-learning with Constraintscited by 17
  22. 2020Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulationcited by 17
  23. 2024FlowPG: Action-constrained Policy Gradient with Normalizing Flowscited by 16
  24. 2025Reinforcement Learning via Implicit Imitation Guidancecited by 12
  25. 2023Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learningcited by 10
  26. 2023Demonstration-Regularized RLcited by 9
  27. 2023Prioritized Soft Q-Decomposition for Lexicographic Reinforcement Learningcited by 5
  28. 2025CHEQ-ing the Box: Safe Variable Impedance Learning for Robotic Polishingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Adaptive Policy Backbone via Shared Network
  2. 2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
  3. 2025Demonstration-Guided Continual Reinforcement Learning in Dynamic Environments
  4. 2026Dynamical Priors as a Training Objective in Reinforcement Learning
  5. 2025Learning to Plan, Planning to Learn: Adaptive Hierarchical RL-MPC for Sample-Efficient Decision Making
  6. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
  7. 2025Accelerated Online Reinforcement Learning using Auxiliary Start State Distributions
  8. 2026MaPP: A Unified Marginalized Posterior-Predictive Framework for Data-Efficient RLVR