# FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models

arXiv: https://arxiv.org/abs/2109.01951 · HTML: https://mlsanity.com/paper/2109.01951/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 4 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [Training Question Answering Models From Synthetic Data](https://mlsanity.com/paper/2002.09599/) (2020)
- [Few-Shot Question Answering by Pretraining Span Selection](https://mlsanity.com/paper/2101.00438/) (2021)
- [Unsupervised Question Answering by Cloze Translation](https://mlsanity.com/paper/1906.04980/) (2019)
- [How Optimal is Greedy Decoding for Extractive Question Answering?](https://mlsanity.com/paper/2108.05857/) (2021)
- [Gendered Pronoun Resolution using BERT and an extractive question answering formulation](https://mlsanity.com/paper/1906.03695/) (2019)

## Built on by (most cited first, 26 of 26)

- [Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models](https://mlsanity.com/paper/2305.01219/) (2023)
- [CLIP-TD: CLIP Targeted Distillation for Vision-Language Tasks](https://mlsanity.com/paper/2201.05729/) (2022)
- [A Survey on Data Augmentation in Large Model Era](https://mlsanity.com/paper/2401.15422/) (2024)
- [Multimodal Adaptive Distillation for Leveraging Unimodal Encoders for Vision-Language Tasks](https://mlsanity.com/paper/2204.10496/) (2022)
- [Exploring The Landscape of Distributional Robustness for Question Answering Models](https://mlsanity.com/paper/2210.12517/) (2022)
- [Language model acceptability judgements are not always robust to context](https://mlsanity.com/paper/2212.08979/) (2022)
- [Generating Information-Seeking Conversations from Unlabeled Documents](https://mlsanity.com/paper/2205.12609/) (2022)
- [MinPrompt: Graph-based Minimal Prompt Data Augmentation for Few-shot Question Answering](https://mlsanity.com/paper/2310.05007/) (2023)
- [Compositional Task-Oriented Parsing as Abstractive Question Answering](https://mlsanity.com/paper/2205.02068/) (2022)
- [Prompting-based Synthetic Data Generation for Few-Shot Question Answering](https://mlsanity.com/paper/2405.09335/) (2024)
- [Do we still need Human Annotators? Prompting Large Language Models for Aspect Sentiment Quad Prediction](https://mlsanity.com/paper/2502.13044/) (2025)
- [KECP: Knowledge Enhanced Contrastive Prompting for Few-shot Extractive Question Answering](https://mlsanity.com/paper/2205.03071/) (2022)
- [ComplexTempQA:A 100m Dataset for Complex Temporal Question Answering](https://mlsanity.com/paper/2406.04866/) (2024)
- [Gotta: Generative Few-shot Question Answering by Prompt-based Cloze Data Augmentation](https://mlsanity.com/paper/2306.04101/) (2023)
- [ReFusion: Improving Natural Language Understanding with Computation-Efficient Retrieval Representation Fusion](https://mlsanity.com/paper/2401.02993/) (2024)
- [Dynamic Few-Shot Learning for Knowledge Graph Question Answering](https://mlsanity.com/paper/2407.01409/) (2024)
- [Redefining Proactivity for Information Seeking Dialogue](https://mlsanity.com/paper/2410.15297/) (2024)
- [From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Reader](https://mlsanity.com/paper/2212.04755/) (2022)
- [Few-shot Unified Question Answering: Tuning Models or Prompts?](https://mlsanity.com/paper/2305.14569/) (2023)
- [Meta-training with Demonstration Retrieval for Efficient Few-shot Learning](https://mlsanity.com/paper/2307.00119/) (2023)
- [Low-Resource Clickbait Spoiling for Indonesian via Question Answering](https://mlsanity.com/paper/2310.08085/) (2023)
- [Knowledge AI: Fine-tuning NLP Models for Facilitating Scientific Knowledge Extraction and Understanding](https://mlsanity.com/paper/2408.04651/) (2024)
- [CFMS: A Coarse-to-Fine Multimodal Synthesis Framework for Enhanced Tabular Reasoning](https://mlsanity.com/paper/2604.10973/) (2026)
- [Few-shot Multi-hop Question Answering over Knowledge Base](https://mlsanity.com/paper/2112.11909/) (2021)
- [LEARN: A Unified Framework for Multi-Task Domain Adapt Few-Shot Learning](https://mlsanity.com/paper/2412.16275/) (2024)
- [Adapting Small Language Models to Low-Resource Domains: A Case Study in Hindi Tourism QA](https://mlsanity.com/paper/2510.25273/) (2025)

## Abstract (arXiv)

The task of learning from only a few examples (called a few-shot setting) is of key importance and relevance to a real-world setting. For question answering (QA), the current state-of-the-art pre-trained models typically need fine-tuning on tens of thousands of examples to obtain good results. Their performance degrades significantly in a few-shot setting (< 100 examples). To address this, we propose a simple fine-tuning framework that leverages pre-trained text-to-text models and is directly aligned with their pre-training framework. Specifically, we construct the input as a concatenation of the question, a mask token representing the answer span and a context. Given this input, the model is fine-tuned using the same objective as that of its pre-training objective. Through experimental studies on various few-shot configurations, we show that this formulation leads to significant gains on multiple QA benchmarks (an absolute gain of 34.2 F1 points on average when there are only 16 training examples). The gains extend further when used with larger models (Eg:- 72.3 F1 on SQuAD using BART-large with only 32 examples) and translate well to a multilingual setting . On the multilingual TydiQA benchmark, our model outperforms the XLM-Roberta-large by an absolute margin of upto 40 F1 points and an average of 33 F1 points in a few-shot setting (<= 64 training examples). We conduct detailed ablation studies to analyze factors contributing to these gains.