# Safe Value Functions

arXiv: https://arxiv.org/abs/2105.12204 · HTML: https://mlsanity.com/paper/2105.12204/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 81 references that are in the map.
- First posted on arXiv 25 May 2021.

## Builds on (most cited first, 29 of 29)

- [Constrained Policy Optimization](https://mlsanity.com/paper/1705.10528/) (2017)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Reward Constrained Policy Optimization](https://mlsanity.com/paper/1805.11074/) (2018)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Control Barrier Function Based Quadratic Programs for Safety Critical Systems](https://mlsanity.com/paper/1609.06408/) (2016)
- [Hamilton-Jacobi Reachability: A Brief Overview and Recent Advances](https://mlsanity.com/paper/1709.07523/) (2017)
- [A General Safety Framework for Learning-Based Control in Uncertain Robotic Systems](https://mlsanity.com/paper/1705.01292/) (2017)
- [Learning to Walk via Deep Reinforcement Learning](https://mlsanity.com/paper/1812.11103/) (2018)
- [Constrained Reinforcement Learning Has Zero Duality Gap](https://mlsanity.com/paper/1910.13393/) (2019)
- [Learning Control Barrier Functions from Expert Demonstrations](https://mlsanity.com/paper/2004.03315/) (2020)
- [Safe Policies for Reinforcement Learning via Primal-Dual Methods](https://mlsanity.com/paper/1911.09101/) (2019)
- [Safe Exploration for Interactive Machine Learning](https://mlsanity.com/paper/1910.13726/) (2019)
- [Robust Control Barrier-Value Functions for Safety-Critical Control](https://mlsanity.com/paper/2104.02808/) (2021)
- [Constrained Upper Confidence Reinforcement Learning](https://mlsanity.com/paper/2001.09377/) (2020)
- [Learning Stability Certificates from Data](https://mlsanity.com/paper/2008.05952/) (2020)
- [Bayesian Optimization Using Domain Knowledge on the ATRIAS Biped](https://mlsanity.com/paper/1709.06047/) (2017)
- [Safe and Fast Tracking on a Robot Manipulator: Robust MPC and Neural Network Control](https://mlsanity.com/paper/1912.10360/) (2019)
- [Robot Learning with Crash Constraints](https://mlsanity.com/paper/2010.08669/) (2020)
- [Scalable Learning of Safety Guarantees for Autonomous Systems using Hamilton-Jacobi Reachability](https://mlsanity.com/paper/2101.05916/) (2021)
- [Learning Constraints from Demonstrations](https://mlsanity.com/paper/1812.07084/) (2018)
- [Learning Event-triggered Control from Data through Joint Optimization](https://mlsanity.com/paper/2008.04712/) (2020)
- [A Learnable Safety Measure](https://mlsanity.com/paper/1910.02835/) (2019)
- [Reachable Sets for Safe, Real-Time Manipulator Trajectory Design](https://mlsanity.com/paper/2002.01591/) (2020)
- [Walking Control Based on Step Timing Adaptation](https://mlsanity.com/paper/1704.01271/) (2017)
- [DeepQ Stepper: A framework for reactive dynamic walking on uneven terrain](https://mlsanity.com/paper/2010.14834/) (2020)
- [Beyond Basins of Attraction: Quantifying Robustness of Natural Dynamics](https://mlsanity.com/paper/1806.08081/) (2018)
- [Computing controlled invariant sets from data using convex optimization](https://mlsanity.com/paper/1912.03256/) (2019)
- [On exploration requirements for learning safety constraints](https://mlsanity.com/paper/2105.08143/) (2021)
- [Robust Humanoid Locomotion Using Trajectory Optimization and Sample-Efficient Learning](https://mlsanity.com/paper/1907.04616/) (2019)

## Built on by (most cited first, 7 of 7)

- [Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations](https://mlsanity.com/paper/2111.09971/) (2021)
- [Discrete GCBF Proximal Policy Optimization for Multi-agent Safe Optimal Control](https://mlsanity.com/paper/2502.03640/) (2025)
- [Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theory](https://mlsanity.com/paper/2306.04026/) (2023)
- [Solving Multi-Agent Safe Optimal Control with Distributed Epigraph Form MARL](https://mlsanity.com/paper/2504.15425/) (2025)
- [A Risk-Averse Preview-based Q-Learning Algorithm: Application to Highway Driving of Autonomous Vehicles](https://mlsanity.com/paper/2112.03232/) (2021)
- [Viability of Future Actions: Robust Safety in Reinforcement Learning via Entropy Regularization](https://mlsanity.com/paper/2506.10871/) (2025)
- [Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty](https://mlsanity.com/paper/2604.25508/) (2026)

## Abstract (arXiv)

Safety constraints and optimality are important, but sometimes conflicting criteria for controllers. Although these criteria are often solved separately with different tools to maintain formal guarantees, it is also common practice in reinforcement learning to simply modify reward functions by penalizing failures, with the penalty treated as a mere heuristic. We rigorously examine the relationship of both safety and optimality to penalties, and formalize sufficient conditions for safe value functions (SVFs): value functions that are both optimal for a given task, and enforce safety constraints. We reveal this structure by examining when rewards preserve viability under optimal control, and show that there always exists a finite penalty that induces a safe value function. This penalty is not unique, but upper-unbounded: larger penalties do not harm optimality. Although it is often not possible to compute the minimum required penalty, we reveal clear structure of how the penalty, rewards, discount factor, and dynamics interact. This insight suggests practical, theory-guided heuristics to design reward functions for control problems where safety is important.