mlsanityData to 7 Oct 2026

Reinforcement Learning with a Corrupted Reward Channel

arXiv 1705.08417 ↗First posted 23 May 2017cs.AI cs.LG stat.MLcited by 68

Not on the map. Why?

Abstract (arXiv)

No real-world reward function is perfect. Sensory errors and software bugs may result in RL agents observing higher (or lower) rewards than they should. For example, a reinforcement learning agent may prefer states where a sensory error gives it the maximum reward, but where the true reward is actually small. We formalise this problem as a generalised Markov Decision Problem called Corrupt Reward MDP. Traditional RL methods fare poorly in CRMDPs, even under strong simplifying assumptions and when trying to compensate for the possibly corrupt rewards. Two ways around the problem are investigated. First, by giving the agent richer data, such as in inverse reinforcement learning and semi-supervised reinforcement learning, reward corruption stemming from systematic sensory errors may sometimes be completely managed. Second, by using randomisation to blunt the agent's optimisation, reward corruption can be partially managed under some assumptions.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 2; 2021: 6; 2022: 9; 2023: 11; 2024: 6; 2025: 21; 2026: 12.

68 papers.

Sort:

By area:

  1. 2023Let's Verify Step by Stepcited by 3,598
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  3. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  5. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  6. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  7. 2022Defining and Characterizing Reward Hackingcited by 184
  8. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2022Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoningcited by 93
  11. 2025GRPO-LEAD: A Difficulty-Aware Reinforcement Learning Approach for Concise Mathematical Reasoning in Language Modelscited by 81
  12. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
38 more
  1. 2021On the Expressivity of Markov Rewardcited by 71
  2. 2025SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLMcited by 67
  3. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  5. 2023Goodhart's Law in Reinforcement Learningcited by 37
  6. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32
  7. 2025The Traitors: Deception and Trust in Multi-Agent Language Model Simulationscited by 30
  8. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  9. 2021MORAL: Aligning AI with Human Norms through Multi-Objective Reinforced Active Learningcited by 24
  10. 2025MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hackingcited by 19
  11. 2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Modelscited by 18
  12. 2020Lipschitzness Is All You Need To Tame Off-policy Generative Adversarial Imitation Learningcited by 15
  13. 2023STARC: A General Framework For Quantifying Differences Between Reward Functionscited by 15
  14. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  15. 2023Internally Rewarded Reinforcement Learningcited by 14
  16. 2021No-Regret Reinforcement Learning with Heavy-Tailed Rewardscited by 13
  17. 2021Impossibility Results in AI: A Surveycited by 13
  18. 2026An Imperfect Verifier is Good Enough: Learning with Noisy Rewardscited by 11
  19. 2026RewardHackingAgents: Benchmarking Evaluation Integrity for LLM ML-Engineering Agentscited by 10
  20. 2025AI Agents and the Lawcited by 9
  21. 2023Solving Non-Rectangular Reward-Robust MDPs via Frequency Regularizationcited by 8
  22. 2025Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learningcited by 7
  23. 2025Reinforcement Learning with Stochastic Reward Machinescited by 7
  24. 2022Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learningcited by 6
  25. 2025Machine Bullshit: Characterizing the Emergent Disregard for Truth in Large Language Modelscited by 6
  26. 2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 6
  27. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
  28. 2024Reinforcement Learning from LLM Feedback to Counteract Goal Misgeneralizationcited by 5
  29. 2023Mutation Testing of Deep Reinforcement Learning Based on Real Faultscited by 4
  30. 2023Online Estimation and Inference for Robust Policy Evaluation in Reinforcement Learningcited by 4
  31. 2025RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translationcited by 3
  32. 2025Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoningcited by 3
  33. 2023Beyond Expected Return: Accounting for Policy Reproducibility when Evaluating Reinforcement Learning Algorithmscited by 2
  34. 2024RL, but don't do anything I wouldn't docited by 2
  35. 2025Causally-Enhanced Reinforcement Policy Optimizationcited by 2
  36. 2026Alignment Verifiability in Large Language Models: Normative Indistinguishability under Behavioral Evaluationcited by 2
  37. 2026Intelligence as Managed Autonomy: Failure, Escalation, and Governance for Agentic AI Systemscited by 2
  38. 2021Stable deep reinforcement learning method by predicting uncertainty in rewards as a subtaskcited by 1

50 most cited of 68 All 68 (JSON)