Data to 7 Oct 2026

Maximum Entropy RL (Provably) Solves Some Robust RL Problems

arXiv 2103.06257 ↗First posted 10 Mar 2021cs.LG cs.ROcited by 169

Not on the map. Why?

Abstract (arXiv)

Many potential applications of reinforcement learning (RL) require guarantees that the agent will perform well in the face of disturbances to the dynamics or reward function. In this paper, we prove theoretically that maximum entropy (MaxEnt) RL maximizes a lower bound on a robust RL objective, and thus can be used to learn policies that are robust to some disturbances in the dynamics and the reward function. While this capability of MaxEnt RL has been observed empirically in prior work, to the best of our knowledge our work provides the first rigorous proof and theoretical characterization of the MaxEnt RL robust set. While a number of prior robust RL algorithms have been designed to handle similar disturbances to the reward function or dynamics, these methods typically require additional moving parts and hyperparameters on top of a base RL algorithm. In contrast, our results suggest that MaxEnt RL by itself is robust to certain disturbances, without requiring any additional modifications. While this does not imply that MaxEnt RL is the best available robust RL method, MaxEnt RL is a simple robust RL method with appealing formal guarantees.

Key facts. Cited by 169 papers in the map (data through 7 Oct 2026). Builds on 49 of its 85 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

49 of 85 references.

References · 49

2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2015Continuous control with deep reinforcement learningcited by 3,279
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2016Concrete Problems in AI Safetycited by 1,582
2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
2017Constrained Policy Optimizationcited by 928
2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
2017Robust Adversarial Reinforcement Learningcited by 439

37 more

2016The CMA Evolution Strategy: A Tutorialcited by 364
2018Quantifying Generalization in Reinforcement Learningcited by 345
2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
2018Maximum a Posteriori Policy Optimisationcited by 318
2019A Theory of Regularized Markov Decision Processescited by 279
2017Inverse Reward Designcited by 246
2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
2017A unified view of entropy-regularized Markov decision processescited by 174
2018Understanding the impact of entropy on policy optimizationcited by 171
2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
2020Recovery RL: Safe Reinforcement Learning with Learned Recovery Zonescited by 156
2018Learning to Walk via Deep Reinforcement Learningcited by 153
2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
2019Lyapunov-based Safe Policy Optimization for Continuous Controlcited by 140
2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
2013Risk-sensitive Reinforcement Learningcited by 129
2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
2020Learning to be Safe: Deep RL with a Safety Criticcited by 109
2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
2018Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generationcited by 80
2020Munchausen Reinforcement Learningcited by 68
2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
2019Worst Cases Policy Gradientscited by 52
2020Robust Reinforcement Learning via Adversarial training with Langevin Dynamicscited by 45
2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
2020Understanding Learned Reward Functionscited by 37
2019Positive-Unlabeled Reward Learningcited by 36
2018The Many Faces of Exponential Weights in Online Learningcited by 33
2020Leverage the Average: an Analysis of KL Regularization in RLcited by 33
2020Lipschitz Lifelong Reinforcement Learningcited by 23
2019Tsallis Reinforcement Learning: A Unified Framework for Maximum Entropy Reinforcement Learningcited by 18
2020LESS is More: Rethinking Probabilistic Models of Human Behaviorcited by 17
2020Entropic Risk Constrained Soft-Robust Policy Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 21; 2023: 37; 2024: 30; 2025: 47; 2026: 23.

169 papers.

Sort:

By area:

2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 463
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoningcited by 160
2023Amortizing intractable inference in large language modelscited by 94
2024Robust agents learn causal world modelscited by 81
2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
2021CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothingcited by 41
2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 40
2022On the Robustness of Safe Reinforcement Learning under Observational Perturbationscited by 38

38 more

2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignmentcited by 38
2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
2021A Statistical Analysis of Polyak-Ruppert Averaged Q-learningcited by 29
2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
2026Maximum Likelihood Reinforcement Learningcited by 29
2021A Max-Min Entropy Framework for Reinforcement Learningcited by 28
2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
2024Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flowcited by 25
2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
2021Robust Object Detection via Instance-Level Temporal Cycle Confusioncited by 24
2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
2021Learn-to-Race: A Multimodal Control Environment for Autonomous Racingcited by 23
2025RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbationscited by 23
2023Robust Average-Reward Markov Decision Processescited by 22
2024Understanding Reference Policies in Direct Preference Optimizationcited by 22
2022Your Policy Regularizer is Secretly an Adversarycited by 21
2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
2025Synthesis of Model Predictive Control and Reinforcement Learning: Survey and Classificationcited by 19
2024S2AC: Energy-Based Reinforcement Learning with Stein Soft Actor Criticcited by 18
2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RLcited by 17
2025From discrete-time policies to continuous-time diffusion samplers: Asymptotic equivalences and faster trainingcited by 17
2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learningcited by 17
2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
2022Risk-Sensitive Reinforcement Learning with Exponential Criteriacited by 15
2025Horizon Generalization in Reinforcement Learningcited by 15
2024Stabilizing Reinforcement Learning in Differentiable Multiphysics Simulationcited by 14
2022Latent State Marginalization as a Low-cost Approach for Improving Explorationcited by 13
2025QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMscited by 13
2025EntroPIC: Towards Stable Long-Term Training of LLMs via Entropy Stabilization with Proportional-Integral Controlcited by 13
2022Safe Reinforcement Learning From Pixels Using a Stochastic Latent Representationcited by 11
2023Reinforcement Learning with Simple Sequence Priorscited by 11
2026MASPO: Unifying Gradient Utilization, Probability Mass, and Signal Reliability for Robust and Sample-Efficient LLM Reasoningcited by 11
2023Bad Habits: Policy Confounding and Out-of-Trajectory Generalization in RLcited by 10
2026Entropy-Preserving Reinforcement Learningcited by 10
2022Robust Reinforcement Learning with Distributional Risk-averse formulationcited by 9
2023Ask more, know better: Reinforce-Learned Prompt Questions for Decision Making with Large Language Modelscited by 9

50 most cited of 169 All 169 (JSON)