# Value Summation: A Novel Scoring Function for MPC-based Model-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2209.08169 · HTML: https://mlsanity.com/paper/2209.08169/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 16 Sep 2022.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](https://mlsanity.com/paper/1803.00101/) (2018)
- [Value Prediction Network](https://mlsanity.com/paper/1707.03497/) (2017)
- [Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control](https://mlsanity.com/paper/1811.01848/) (2018)
- [A Survey on Model-based Reinforcement Learning](https://mlsanity.com/paper/2206.09328/) (2022)
- [Learning Off-Policy with Online Planning](https://mlsanity.com/paper/2008.10066/) (2020)
- [Feedback Control For Cassie With Deep Reinforcement Learning](https://mlsanity.com/paper/1803.05580/) (2018)
- [Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.13842/) (2021)
- [Deep Value Model Predictive Control](https://mlsanity.com/paper/1910.03358/) (2019)
- [Iterative Reinforcement Learning Based Design of Dynamic Locomotion Skills for Cassie](https://mlsanity.com/paper/1903.09537/) (2019)
- [Solving Challenging Dexterous Manipulation Tasks With Trajectory Optimisation and Reinforcement Learning](https://mlsanity.com/paper/2009.05104/) (2020)

## Built on by (most cited first, 1 of 1)

- [Sample-efficient Real-time Planning with Curiosity Cross-Entropy Method and Contrastive Learning](https://mlsanity.com/paper/2303.03787/) (2023)

## Abstract (arXiv)

This paper proposes a novel scoring function for the planning module of MPC-based reinforcement learning methods to address the inherent bias of using the reward function to score trajectories. The proposed method enhances the learning efficiency of existing MPC-based MBRL methods using the discounted sum of values. The method utilizes optimal trajectories to guide policy learning and updates its state-action value function based on real-world and augmented onboard data. The learning efficiency of the proposed method is evaluated in selected MuJoCo Gym environments as well as in learning locomotion skills for a simulated model of the Cassie robot. The results demonstrate that the proposed method outperforms the current state-of-the-art algorithms in terms of learning efficiency and average reward return.