# Learning Answer Generation using Supervision from Automatic Question Answering Evaluators

arXiv: https://arxiv.org/abs/2305.15344 · HTML: https://mlsanity.com/paper/2305.15344/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 30 of its 41 references that are in the map.
- First posted on arXiv 24 May 2023.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [COMET: A Neural Framework for MT Evaluation](https://mlsanity.com/paper/2009.09025/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [On Decoding Strategies for Neural Text Generators](https://mlsanity.com/paper/2203.15721/) (2022)
- [TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection](https://mlsanity.com/paper/1911.04118/) (2019)
- [Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluation](https://mlsanity.com/paper/2202.07654/) (2022)
- [Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2112.08688/) (2021)
- [R2-D2: A Modular Baseline for Open-Domain Question Answering](https://mlsanity.com/paper/2109.03502/) (2021)
- [Joint Learning of Answer Selection and Answer Summary Generation in Community Question Answering](https://mlsanity.com/paper/1911.09801/) (2019)
- [What's in a Name? Answer Equivalence For Open-Domain Question Answering](https://mlsanity.com/paper/2109.05289/) (2021)
- [Adaptations of ROUGE and BLEU to Better Evaluate Machine Reading Comprehension Task](https://mlsanity.com/paper/1806.03578/) (2018)
- [Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering](https://mlsanity.com/paper/2109.07009/) (2021)
- [Attention-guided Generative Models for Extractive Question Answering](https://mlsanity.com/paper/2110.06393/) (2021)
- [Answer Generation for Retrieval-based Question Answering Systems](https://mlsanity.com/paper/2106.00955/) (2021)
- [Joint Models for Answer Verification in Question Answering Systems](https://mlsanity.com/paper/2107.04217/) (2021)
- [AVA: an Automatic eValuation Approach to Question Answering Systems](https://mlsanity.com/paper/2005.00705/) (2020)
- [Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection](https://mlsanity.com/paper/2205.10455/) (2022)
- [Paragraph-based Transformer Pre-training for Multi-Sentence Inference](https://mlsanity.com/paper/2205.01228/) (2022)
- [Cross-Lingual Open-Domain Question Answering with Answer Sentence Generation](https://mlsanity.com/paper/2110.07150/) (2021)
- [Knowledge Transfer from Answer Ranking to Answer Generation](https://mlsanity.com/paper/2210.12865/) (2022)

## Built on by (most cited first, 3 of 3)

- [PACE: Improving Prompt with Actor-Critic Editing for Large Language Model](https://mlsanity.com/paper/2308.10088/) (2023)
- [SQUARE: Automatic Question Answering Evaluation using Multiple Positive and Negative References](https://mlsanity.com/paper/2309.12250/) (2023)
- [Datasets for Multilingual Answer Sentence Selection](https://mlsanity.com/paper/2406.10172/) (2024)

## Abstract (arXiv)

Recent studies show that sentence-level extractive QA, i.e., based on Answer Sentence Selection (AS2), is outperformed by Generation-based QA (GenQA) models, which generate answers using the top-k answer sentences ranked by AS2 models (a la retrieval-augmented generation style). In this paper, we propose a novel training paradigm for GenQA using supervision from automatic QA evaluation models (GAVA). Specifically, we propose three strategies to transfer knowledge from these QA evaluation models to a GenQA model: (i) augmenting training data with answers generated by the GenQA model and labelled by GAVA (either statically, before training, or (ii) dynamically, at every training epoch); and (iii) using the GAVA score for weighting the generator loss during the learning of the GenQA model. We evaluate our proposed methods on two academic and one industrial dataset, obtaining a significant improvement in answering accuracy over the previous state of the art.