mlsanityData to 7 Oct 2026

Optimizing Neurorobot Policy under Limited Demonstration Data through Preference Regret

arXiv 2604.03523 ↗First posted 4 Apr 2026cs.RO cs.AI cs.CVcited by 0

On the map: Offline RL and imitation learning from data → Robot learning from demonstration

Abstract (arXiv)

Robot reinforcement learning from demonstrations (RLfD) assumes that expert data is abundant; this is usually unrealistic in the real world given data scarcity as well as high collection cost. Furthermore, imitation learning algorithms assume that the data is independently and identically distributed, which ultimately results in poorer performance as gradual errors emerge and compound within test-time trajectories. We address these issues by introducing the "master your own expertise" (MYOE) framework, a self-imitation framework that enables robotic agents to learn complex behaviors from limited demonstration data samples. Inspired by human perception and action, we propose and design what we call the queryable mixture-of-preferences state space model (QMoP-SSM), which estimates the desired goal at every time step. These desired goals are used in computing the "preference regret", which is used to optimize the robot control policy. Our experiments demonstrate the robustness, adaptability, and out-of-sample performance of our agent compared to other state-of-the-art RLfD schemes. The GitHub repository that supports this work can be found at: https://github.com/rxng8/neurorobot-preference-regret-learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Robot learning from demonstration, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 69 references that are in the map. First posted on arXiv 4 Apr 2026.

Builds on

32 of 69 references.

References · 32

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  5. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018World Modelscited by 1,444
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2020Mastering Atari with Discrete World Modelscited by 990
20 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  3. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  4. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  5. 2019Exploring the Limitations of Behavior Cloning for Autonomous Drivingcited by 343
  6. 2015Deep Kalman Filterscited by 163
  7. 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
  8. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  9. 2018Probabilistic Recurrent State-Space Modelscited by 64
  10. 2019Deep Active Inference as Variational Policy Gradientscited by 52
  11. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  12. 2020Whence the Expected Free Energy?cited by 40
  13. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  14. 2020Self-Imitation Advantage Learningcited by 15
  15. 2023Mortal Computation: A Foundation for Biomimetic Intelligencecited by 15
  16. 2019Bayesian policy selection using active inferencecited by 12
  17. 2021Backprop-Free Reinforcement Learning with Active Neural Generative Codingcited by 12
  18. 2023Adversarial Imitation Learning from Visual Observations using Latent Informationcited by 10
  19. 2024R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Modelscited by 10
  20. 2021Prior Preference Learning from Experts:Designing a Reward with Active Inferencecited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Imitation Learning by Estimating Expertise of Demonstrators
  2. 2022Fast Lifelong Adaptive Inverse Reinforcement Learning from Demonstrations
  3. 2025Fine-tuning Behavioral Cloning Policies with Preference-Based Reinforcement Learning
  4. 2025Search-Based Credit Assignment for Offline Preference-Based Reinforcement Learning
  5. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learning
  6. 2022Autonomous Assessment of Demonstration Sufficiency via Bayesian Inverse Reinforcement Learning
  7. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasets
  8. 2018An Algorithmic Perspective on Imitation Learning