mlsanityData to 7 Oct 2026

Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedback

arXiv 1805.01252 ↗First posted 3 May 2018cs.CL cs.LG stat.MLcited by 32

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Counterfactual learning from human bandit feedback describes a scenario where user feedback on the quality of outputs of a historic system is logged and used to improve a target system. We show how to apply this learning framework to neural semantic parsing. From a machine learning perspective, the key challenge lies in a proper reweighting of the estimator so as to avoid known degeneracies in counterfactual learning, while still being applicable to stochastic gradient optimization. To conduct experiments with human users, we devise an easy-to-use interface to collect human feedback on semantic parses. Our work is the first to show that semantic parsers can be improved significantly by counterfactual learning from logged human feedback data.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 34 references that are in the map. First posted on arXiv 3 May 2018.

Builds on

15 of 34 references.

References · 15

  1. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  3. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  5. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  6. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  7. 2016Language to Logical Form with Neural Attentioncited by 158
  8. 2016Data Recombination for Neural Semantic Parsingcited by 111
  9. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 101
  10. 2017Learning a Neural Semantic Parser from User Feedbackcited by 98
  11. 2017Nematus: a Toolkit for Neural Machine Translationcited by 16
  12. 2016Learning a Natural Language Interface with Neural Programmercited by 14
3 more
  1. 2017Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translationcited by 13
  2. 2016Coupling Distributed and Symbolic Execution for Natural Language Queriescited by 9
  3. 2017Counterfactual Learning for Machine Translation: Degeneracies and Solutionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 2; 2021: 6; 2022: 4; 2023: 12; 2024: 4; 2025: 2; 2026: 1.

32 papers.

Sort:

By area:

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  4. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  5. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  6. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  7. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  8. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 53
  9. 2023Symbol-LLM: Towards Foundational Symbol-centric Interface For Large Language Modelscited by 38
  10. 2024M-RAG: Reinforcing Large Language Model Performance through Retrieval-Augmented Generation with Multiple Partitionscited by 38
  11. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  12. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
20 more
  1. 2023Sample Efficient Preference Alignment in LLMs via Active Explorationcited by 18
  2. 2024RePair: Automated Program Repair with Process-based Feedbackcited by 16
  3. 2022Simulating Bandit Learning from User Feedback for Extractive Question Answeringcited by 14
  4. 2020Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Taskscited by 12
  5. 2023ChatGPT vs State-of-the-Art Models: A Benchmarking Study in Keyphrase Generation Taskcited by 12
  6. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  7. 2021Total Recall: a Customized Continual Learning Method for Neural Semantic Parserscited by 9
  8. 2023Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMapcited by 9
  9. 2024Multimodal Query Suggestion with Multi-Agent Reinforcement Learning from Human Feedbackcited by 9
  10. 2023Multi-Source Test-Time Adaptation as Dueling Bandits for Extractive Question Answeringcited by 6
  11. 2018Counterfactual Learning from Human Proofreading Feedback for Semantic Parsingcited by 4
  12. 2023Kernelized Offline Contextual Dueling Banditscited by 4
  13. 2021One Semantic Parser to Parse Them All: Sequence to Sequence Multi-Task Learning on Semantic Parsing Datasetscited by 3
  14. 2021Cogment: Open Source Framework For Distributed Multi-actor Training, Deployment & Operationscited by 3
  15. 2022A Survey on Artificial Intelligence for Source Code: A Dialogue Systems Perspectivecited by 3
  16. 2023Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compressioncited by 3
  17. 2021Error-Aware Interactive Semantic Parsing of OpenStreetMapcited by 1
  18. 2025Text-to-TrajVis: Enabling Trajectory Data Visualizations from Natural Language Questionscited by 1
  19. 2025OsmT: Bridging OpenStreetMap Queries and Natural Language with Open-source Tag-aware Language Modelscited by 0
  20. 2026PleaSQLarify: Visual Pragmatic Repair for Natural Language Database Queryingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020An Imitation Game for Learning Semantic Parsers from User Interaction
  2. 2025Counterfactual reasoning: an analysis of in-context emergence
  3. 2022CORE: A Retrieve-then-Edit Framework for Counterfactual Data Generation
  4. 2026DeepSWIP: Quotient-WMC Counterfactuals for Neural Probabilistic Logic Programs
  5. 2023Prompting Large Language Models for Counterfactual Generation: An Empirical Study
  6. 2025Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks
  7. 2019Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs
  8. 2023Aligning Neural Machine Translation Models: Human Feedback in Training and Inference