mlsanityData to 7 Oct 2026

Simulating Bandit Learning from User Feedback for Extractive Question Answering

arXiv 2203.10079 ↗First posted 18 Mar 2022cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

We study learning from user feedback for extractive question answering by simulating feedback using supervised data. We cast the problem as contextual bandit learning, and analyze the characteristics of several learning scenarios with focus on reducing data annotation. We show that systems initially trained on a small number of examples can dramatically improve given feedback from users on model-predicted answers, and that one can use existing datasets to deploy systems in new domains without any annotation, but instead improving the system on-the-fly via user feedback.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 18 Mar 2022.

Builds on

35 of 53 references.

References · 35

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  8. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  9. 2018QuAC : Question Answering in Contextcited by 379
  10. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  11. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  12. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
23 more
  1. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  2. 2017SearchQA: A New Q&A Dataset Augmented with Context from a Search Enginecited by 221
  3. 2019MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehensioncited by 183
  4. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  5. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  6. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  7. 2017Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedbackcited by 71
  8. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  9. 2018A Contextual Bandit Bake-offcited by 64
  10. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  11. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  12. 2020QED: A Framework and Dataset for Explanations in Question Answeringcited by 55
  13. 2018Improving a Neural Semantic Parser by Counterfactual Learning from Human Bandit Feedbackcited by 32
  14. 2020An Imitation Game for Learning Semantic Parsers from User Interactioncited by 23
  15. 2019Domain-agnostic Question-Answering with Adversarial Trainingcited by 22
  16. 2021Contrastive Domain Adaptation for Question Answering using Limited Text Corporacited by 20
  17. 2021Continual Learning for Grounded Instruction Generation by Observing Human Following Behaviorcited by 19
  18. 2017Counterfactual Learning from Bandit Feedback under Deterministic Logging: A Case Study in Statistical Machine Translationcited by 13
  19. 2020Learning a Cost-Effective Annotation Policy for Question Answeringcited by 9
  20. 2021Back-Training excels Self-Training at Unsupervised Domain Adaptation of Question Generation and Passage Retrievalcited by 8
  21. 2021Online Learning Meets Machine Translation Evaluation: Finding the Best Systems with the Least Human Effortcited by 7
  22. 2017A Shared Task on Bandit Learning for Machine Translationcited by 6
  23. 2020Improving Conversational Question Answering Systems after Deployment using Feedback-Weighted Learningcited by 4

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 7; 2024: 2; 2025: 4.

14 papers.

Sort:

By area:

  1. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputscited by 104
  2. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  3. 2024When is Tree Search Useful for LLM Planning? It Depends on the Discriminatorcited by 46
  4. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  5. 2022Continual Learning for Instruction Following from Realtime Feedbackcited by 21
  6. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  7. 2023Continually Improving Extractive QA via Human Feedbackcited by 11
  8. 2023Let Me Teach You: Pedagogical Foundations of Feedback for Language Modelscited by 9
  9. 2023Constructive Large Language Models Alignment with Diverse Feedbackcited by 9
  10. 2023Robust Question Answering against Distribution Shifts with Test-Time Adaptation: An Empirical Studycited by 8
  11. 2023Multi-Source Test-Time Adaptation as Dueling Bandits for Extractive Question Answeringcited by 6
  12. 2025Knowing What's Missing: Assessing Information Sufficiency in Question Answeringcited by 3
2 more
  1. 2023What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perceptioncited by 2
  2. 2025One Framework to Rule Them All: Unifying RL-Based and RL-Free Methods in RLHFcited by 2