# Argumentative Reward Learning: Reasoning About Human Preferences

arXiv: https://arxiv.org/abs/2209.14010 · HTML: https://mlsanity.com/paper/2209.14010/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 28 Sep 2022.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward modeling and RLHF](https://mlsanity.com/alignment/reward-modeling-and-rlhf/)

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [Scalable agent alignment via reward modeling: a research direction](https://mlsanity.com/paper/1811.07871/) (2018)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Cooperative Inverse Reinforcement Learning](https://mlsanity.com/paper/1606.03137/) (2016)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Deep Learning: A Critical Appraisal](https://mlsanity.com/paper/1801.00631/) (2018)
- [Learning Language Games through Interaction](https://mlsanity.com/paper/1606.02447/) (2016)
- [Visual Concept-Metaconcept Learning](https://mlsanity.com/paper/2002.01464/) (2020)
- [On the Acceptability of Arguments in Preference-Based Argumentation](https://mlsanity.com/paper/1301.7358/) (2013)
- [Parenting: Safe Reinforcement Learning from Human Input](https://mlsanity.com/paper/1902.06766/) (2019)
- [An Extensible Interactive Interface for Agent Design](https://mlsanity.com/paper/1906.02641/) (2019)

## Built on by (most cited first, 2 of 2)

- [REFINER: Reasoning Feedback on Intermediate Representations](https://mlsanity.com/paper/2304.01904/) (2023)
- [Neurosymbolic Learning for Inference-Time Argumentation](https://mlsanity.com/paper/2605.20098/) (2026)

## Abstract (arXiv)

We define a novel neuro-symbolic framework, argumentative reward learning, which combines preference-based argumentation with existing approaches to reinforcement learning from human feedback. Our method improves prior work by generalising human preferences, reducing the burden on the user and increasing the robustness of the reward model. We demonstrate this with a number of experiments.