# Reward Reports for Reinforcement Learning

arXiv: https://arxiv.org/abs/2204.10817 · HTML: https://mlsanity.com/paper/2204.10817/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 108 references that are in the map.
- First posted on arXiv 22 Apr 2022.

## Builds on (most cited first, 30 of 30)

- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure](https://mlsanity.com/paper/2010.13561/) (2020)
- [Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches](https://mlsanity.com/paper/1912.05100/) (2019)
- [Simple random search provides a competitive approach to reinforcement learning](https://mlsanity.com/paper/1803.07055/) (2018)
- [Flow: A Modular Learning Framework for Mixed Autonomy Traffic](https://mlsanity.com/paper/1710.05465/) (2017)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)
- [Algorithms for Fairness in Sequential Decision Making](https://mlsanity.com/paper/1901.08568/) (2019)
- [User Tampering in Reinforcement Learning Recommender Systems](https://mlsanity.com/paper/2109.04083/) (2021)
- [Ethics Sheets for AI Tasks](https://mlsanity.com/paper/2107.01183/) (2021)
- [Data Readiness Report](https://mlsanity.com/paper/2010.07213/) (2020)
- [A Methodology for Creating AI FactSheets](https://mlsanity.com/paper/2006.13796/) (2020)
- [Towards Traceability in Data Ecosystems using a Bill of Materials Model](https://mlsanity.com/paper/1904.04253/) (2019)
- [T-RECS: A Simulation Tool to Study the Societal Impact of Recommender Systems](https://mlsanity.com/paper/2107.08959/) (2021)
- [Choices, Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systems](https://mlsanity.com/paper/2202.05716/) (2022)
- [Towards Content Provider Aware Recommender Systems: A Simulation Study on the Interplay between User and Provider Utilities](https://mlsanity.com/paper/2105.02377/) (2021)
- [A Map of Bandits for E-commerce](https://mlsanity.com/paper/2107.00680/) (2021)
- [AI Development for the Public Interest: From Abstraction Traps to Sociotechnical Risks](https://mlsanity.com/paper/2102.04255/) (2021)
- [Axes for Sociotechnical Inquiry in AI Research](https://mlsanity.com/paper/2105.06551/) (2021)
- [Multisource AI Scorecard Table for System Evaluation](https://mlsanity.com/paper/2102.03985/) (2021)

## Built on by (most cited first, 18 of 18)

- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [Frontier AI Regulation: Managing Emerging Risks to Public Safety](https://mlsanity.com/paper/2307.03718/) (2023)
- [Visibility into AI Agents](https://mlsanity.com/paper/2401.13138/) (2024)
- [Beyond Preferences in AI Alignment](https://mlsanity.com/paper/2408.16984/) (2024)
- [The History and Risks of Reinforcement Learning and Human Feedback](https://mlsanity.com/paper/2310.13595/) (2023)
- [Aligning to Social Norms and Values in Interactive Narratives](https://mlsanity.com/paper/2205.01975/) (2022)
- [The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systems](https://mlsanity.com/paper/2602.17753/) (2026)
- [IDs for AI Systems](https://mlsanity.com/paper/2406.12137/) (2024)
- [The AI Agent Index](https://mlsanity.com/paper/2502.01635/) (2025)
- [Acceptable Use Policies for Foundation Models](https://mlsanity.com/paper/2409.09041/) (2024)
- [Accounting for AI and Users Shaping One Another: The Role of Mathematical Models](https://mlsanity.com/paper/2404.12366/) (2024)
- [Responsible Reporting for Frontier AI Development](https://mlsanity.com/paper/2404.02675/) (2024)
- [Reward Model Interpretability via Optimal and Pessimal Tokens](https://mlsanity.com/paper/2506.07326/) (2025)
- [Optimization's Neglected Normative Commitments](https://mlsanity.com/paper/2305.17465/) (2023)
- [FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines](https://mlsanity.com/paper/2307.15475/) (2023)
- [Dynamic Documentation for AI Systems](https://mlsanity.com/paper/2303.10854/) (2023)
- [A case for data valuation transparency via DValCards](https://mlsanity.com/paper/2506.23349/) (2025)

## Abstract (arXiv)

Building systems that are good for society in the face of complex societal effects requires a dynamic approach. Recent approaches to machine learning (ML) documentation have demonstrated the promise of discursive frameworks for deliberation about these complexities. However, these developments have been grounded in a static ML paradigm, leaving the role of feedback and post-deployment performance unexamined. Meanwhile, recent work in reinforcement learning has shown that the effects of feedback and optimization objectives on system behavior can be wide-ranging and unpredictable. In this paper we sketch a framework for documenting deployed and iteratively updated learning systems, which we call Reward Reports. Taking inspiration from various contributions to the technical literature on reinforcement learning, we outline Reward Reports as living documents that track updates to design choices and assumptions behind what a particular automated system is optimizing for. They are intended to track dynamic phenomena arising from system deployment, rather than merely static properties of models or data. After presenting the elements of a Reward Report, we discuss a concrete example: Meta's BlenderBot 3 chatbot. Several others for game-playing (DeepMind's MuZero), content recommendation (MovieLens), and traffic control (Project Flow) are included in the appendix.