mlsanityData to 7 Oct 2026

Preferences Implicit in the State of the World

arXiv 1902.04198 ↗First posted 12 Feb 2019cs.LG cs.AI stat.MLcited by 33

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to fill in the blanks. We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties. We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized. Our code can be found at https://github.com/HumanCompatibleAI/rlsp.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Feb 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 7; 2023: 7; 2024: 7; 2025: 1; 2026: 1.

33 papers.

Sort:

By area:

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  2. 2021Unsolved Problems in ML Safetycited by 291
  3. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  4. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  5. 2024AI Alignment with Changing and Influenceable Reward Functionscited by 33
  6. 2023Learning and Adapting Agile Locomotion Skills by Transferring Experiencecited by 25
  7. 2023TASRA: a Taxonomy and Analysis of Societal-Scale Risks from AIcited by 25
  8. 2021The MineRL BASALT Competition on Learning from Human Feedbackcited by 24
  9. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 23
  10. 2022Humans are not Boltzmann Distributions: Challenges and Opportunities for Modelling Human Feedback and Interaction in Reinforcement Learningcited by 22
  11. 2021Choice Set Misspecification in Reward Inferencecited by 13
  12. 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
21 more
  1. 2021Preference-based Learning of Reward Function Featurescited by 9
  2. 2021Combining Reward Information from Multiple Sourcescited by 7
  3. 2022Preprocessing Reward Functions for Interpretabilitycited by 7
  4. 2024Human-AI Safety: A Descendant of Generative AI and Control Systems Safetycited by 7
  5. 2026Analyzing and Improving Chain-of-Thought Monitorability Through Information Theorycited by 7
  6. 2022Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformerscited by 6
  7. 2024The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regretcited by 6
  8. 2022Retrospective on the 2021 BASALT Competition on Learning from Human Feedbackcited by 5
  9. 2021Learning What To Do by Simulating the Pastcited by 4
  10. 2023Inferring the Future by Imagining the Pastcited by 4
  11. 2024Dynamic Normativity: Necessary and Sufficient Conditions for Value Alignmentcited by 3
  12. 2021Mitigating Negative Side Effects via Environment Shapingcited by 2
  13. 2023Anticipatory Planning: Improving Long-Lived Planning by Estimating Expected Cost of Future Taskscited by 2
  14. 2023STRAPPER: Preference-based Reinforcement Learning via Self-training Augmentation and Peer Regularizationcited by 1
  15. 2025Learning from Preferences and Mixed Demonstrations in General Settingscited by 1
  16. 2021Challenges for Using Impact Regularizers to Avoid Negative Side Effectscited by 0
  17. 2023Safe MDP Planning by Learning Temporal Patterns of Undesirable Trajectories and Averting Negative Side Effectscited by 0
  18. 2023Risk-averse Batch Active Inverse Reward Designcited by 0
  19. 2024Extinction Risks from AI: Invisible to Science?cited by 0
  20. 2024Safe Reinforcement Learning with Learned Non-Markovian Safety Constraintscited by 0
  21. 2024Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Gamescited by 0