# Counterfactual Learning from Human Proofreading Feedback for Semantic Parsing

arXiv: https://arxiv.org/abs/1811.12239 · HTML: https://mlsanity.com/paper/1811.12239/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 12 of its 27 references that are in the map.
- First posted on arXiv 29 Nov 2018.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Human values and AI feedback alignment](https://mlsanity.com/alignment/human-values-and-ai-feedback-alignment/)

## Builds on (most cited first, 12 of 12)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Doubly Robust Off-policy Value Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1511.03722/) (2015)
- [Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learning](https://mlsanity.com/paper/1604.00923/) (2016)
- [Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision](https://mlsanity.com/paper/1611.00020/) (2016)
- [Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback](https://mlsanity.com/paper/1805.01252/) (2018)
- [Nematus: a Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1703.04357/) (2017)
- [Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translation](https://mlsanity.com/paper/1707.09118/) (2017)
- [Coupling Distributed and Symbolic Execution for Natural Language Queries](https://mlsanity.com/paper/1612.02741/) (2016)
- [Counterfactual Learning for Machine Translation: Degeneracies and Solutions](https://mlsanity.com/paper/1711.08621/) (2017)

## Built on by (most cited first, 4 of 4)

- [Putting Humans in the Natural Language Processing Loop: A Survey](https://mlsanity.com/paper/2103.04044/) (2021)
- [SumRecom: A Personalized Summarization Approach by Learning from Users' Feedback](https://mlsanity.com/paper/2408.07294/) (2024)
- [One Semantic Parser to Parse Them All: Sequence to Sequence Multi-Task Learning on Semantic Parsing Datasets](https://mlsanity.com/paper/2106.04476/) (2021)
- [A Personalized Reinforcement Learning Summarization Service for Learning Structure from Unstructured Data](https://mlsanity.com/paper/2307.05696/) (2023)

## Abstract (arXiv)

In semantic parsing for question-answering, it is often too expensive to collect gold parses or even gold answers as supervision signals. We propose to convert model outputs into a set of human-understandable statements which allow non-expert users to act as proofreaders, providing error markings as learning signals to the parser. Because model outputs were suggested by a historic system, we operate in a counterfactual, or off-policy, learning setup. We introduce new estimators which can effectively leverage the given feedback and which avoid known degeneracies in counterfactual learning, while still being applicable to stochastic gradient optimization for neural semantic parsing. Furthermore, we discuss how our feedback collection method can be seamlessly integrated into deployed virtual personal assistants that embed a semantic parser. Our work is the first to show that semantic parsers can be improved significantly by counterfactual learning from logged human feedback data.