Optimization Algorithm for Feedback and Feedforward Policies towards Robot Control Robust to Sensing Failures
Not on the map. Why?
Abstract (arXiv)
Model-free or learning-based control, in particular, reinforcement learning (RL), is expected to be applied for complex robotic tasks. Traditional RL requires a policy to be optimized is state-dependent, that means, the policy is a kind of feedback (FB) controllers. Due to the necessity of correct state observation in such a FB controller, it is sensitive to sensing failures. To alleviate this drawback of the FB controllers, feedback error learning integrates one of them with a feedforward (FF) controller. RL can be improved by dealing with the FB/FF policies, but to the best of our knowledge, a methodology for learning them in a unified manner has not been developed. In this paper, we propose a new optimization problem for optimizing both the FB/FF policies simultaneously. Inspired by control as inference, the optimization problem considers minimization/maximization of divergences between trajectory, predicted by the composed policy and a stochastic dynamics model, and optimal/non-optimal trajectories. By approximating the stochastic dynamics model using variational method, we naturally derive a regularization between the FB/FF policies. In numerical simulations and a robot experiment, we verified that the proposed method can stably optimize the composed policy even with the different learning law from the traditional RL. In addition, we demonstrated that the FF policy is robust to the sensing failures and can hold the optimal motion. Attached video is also uploaded on youtube: https://youtu.be/zLL4uXIRmrE
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 17 of its 54 references that are in the map. First posted on arXiv 1 Apr 2021.
Builds on
17 of 54 references.
References · 17
- 2016Layer Normalizationcited by 5,579
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
- 2016OpenAI Gymcited by 1,684
- 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
- 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
- 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
- 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
- 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
- 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
- 2014Recurrent Neural Network Regularizationcited by 189
- 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 61
- 2016Reinforcement Learning of POMDPs using Spectral Methodscited by 51
5 more
- 2020How to Train Your Differentiable Filtercited by 23
- 2020t-Soft Update of Target Network for Deep Reinforcement Learningcited by 16
- 2020Proximal Policy Optimization with Relative Pearson Divergencecited by 9
- 2020Adaptive and Multiple Time-scale Eligibility Traces for Online Deep Reinforcement Learningcited by 4
- 2020Towards Deep Robot Learning with Optimizer applicable to Non-stationary Problemscited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.