# APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning

arXiv: https://arxiv.org/abs/1808.09658 · HTML: https://mlsanity.com/paper/1808.09658/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2018.

## Built on by (most cited first, 9 of 9)

- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback](https://mlsanity.com/paper/2303.05453/) (2023)
- [Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation](https://mlsanity.com/paper/2305.00955/) (2023)
- [The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values](https://mlsanity.com/paper/2310.07629/) (2023)
- [Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback](https://mlsanity.com/paper/2204.05512/) (2022)
- [Advances in Preference-based Reinforcement Learning: A Review](https://mlsanity.com/paper/2408.11943/) (2024)
- [Multi-Source Test-Time Adaptation as Dueling Bandits for Extractive Question Answering](https://mlsanity.com/paper/2306.06779/) (2023)
- [Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework](https://mlsanity.com/paper/2411.11761/) (2024)
- [SumRecom: A Personalized Summarization Approach by Learning from Users' Feedback](https://mlsanity.com/paper/2408.07294/) (2024)

## Abstract (arXiv)

We propose a method to perform automatic document summarisation without using reference summaries. Instead, our method interactively learns from users' preferences. The merit of preference-based interactive summarisation is that preferences are easier for users to provide than reference summaries. Existing preference-based interactive learning methods suffer from high sample complexity, i.e. they need to interact with the oracle for many rounds in order to converge. In this work, we propose a new objective function, which enables us to leverage active learning, preference learning and reinforcement learning techniques in order to reduce the sample complexity. Both simulation and real-user experiments suggest that our method significantly advances the state of the art. Our source code is freely available at https://github.com/UKPLab/emnlp2018-april.