# ProQA: Structural Prompt-based Pre-training for Unified Question Answering

arXiv: https://arxiv.org/abs/2205.04040 · HTML: https://mlsanity.com/paper/2205.04040/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 61 references that are in the map.
- First posted on arXiv 9 May 2022.

## Builds on (most cited first, 30 of 46)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering](https://mlsanity.com/paper/1809.02789/) (2018)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [The NarrativeQA Reading Comprehension Challenge](https://mlsanity.com/paper/1712.07040/) (2017)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [NewsQA: A Machine Comprehension Dataset](https://mlsanity.com/paper/1611.09830/) (2016)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [WARP: Word-level Adversarial ReProgramming](https://mlsanity.com/paper/2101.00121/) (2021)
- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension](https://mlsanity.com/paper/1902.00164/) (2019)
- [OpenPrompt: An Open-source Framework for Prompt-learning](https://mlsanity.com/paper/2111.01998/) (2021)
- [Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning](https://mlsanity.com/paper/1908.05803/) (2019)
- [Few-Shot Question Answering by Pretraining Span Selection](https://mlsanity.com/paper/2101.00438/) (2021)
- [Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?](https://mlsanity.com/paper/2106.01561/) (2021)

## Built on by (most cited first, 21 of 21)

- [DiscoPrompt: Path Prediction Prompt Tuning for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2305.03973/) (2023)
- [DUnE: Dataset for Unified Editing](https://mlsanity.com/paper/2311.16087/) (2023)
- [UnifiedABSA: A Unified ABSA Framework Based on Multi-task Instruction Tuning](https://mlsanity.com/paper/2211.10986/) (2022)
- [Interactive Question Answering Systems: Literature Review](https://mlsanity.com/paper/2209.01621/) (2022)
- [Long-Tailed Question Answering in an Open World](https://mlsanity.com/paper/2305.06557/) (2023)
- [REVEAL: Robust Evolution of Vision-Language Models for Explainable AI-Video Detection](https://mlsanity.com/paper/2502.14994/) (2025)
- [Improving Task Generalization via Unified Schema Prompt](https://mlsanity.com/paper/2208.03229/) (2022)
- [Domain Incremental Lifelong Learning in an Open World](https://mlsanity.com/paper/2305.06555/) (2023)
- [It Is Not About What You Say, It Is About How You Say It: A Surprisingly Simple Approach for Improving Reading Comprehension](https://mlsanity.com/paper/2406.16779/) (2024)
- [Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers](https://mlsanity.com/paper/2210.11265/) (2022)
- [Improving Cross-task Generalization of Unified Table-to-text Models with Compositional Task Configurations](https://mlsanity.com/paper/2212.08780/) (2022)
- [Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts](https://mlsanity.com/paper/2305.07019/) (2023)
- [From Cloze to Comprehension: Retrofitting Pre-trained Masked Language Model to Pre-trained Machine Reader](https://mlsanity.com/paper/2212.04755/) (2022)
- [Evaluating Prompt-based Question Answering for Object Prediction in the Open Research Knowledge Graph](https://mlsanity.com/paper/2305.12900/) (2023)
- [Few-shot Unified Question Answering: Tuning Models or Prompts?](https://mlsanity.com/paper/2305.14569/) (2023)
- [LabelPrompt: Effective Prompt-based Learning for Relation Classification](https://mlsanity.com/paper/2302.08068/) (2023)
- [UKP-SQuARE v3: A Platform for Multi-Agent QA Research](https://mlsanity.com/paper/2303.18120/) (2023)
- [Prompt Guided Copy Mechanism for Conversational Question Answering](https://mlsanity.com/paper/2308.03422/) (2023)
- [Chatbot is Not All You Need: Information-rich Prompting for More Realistic Responses](https://mlsanity.com/paper/2312.16233/) (2023)
- [Multi-granular Training Strategies for Robust Multi-hop Reasoning Over Noisy and Heterogeneous Knowledge Sources](https://mlsanity.com/paper/2502.05944/) (2025)
- [How Many Thoughts Can a Vector Hold? The Capacity of Reasoning by Superposition](https://mlsanity.com/paper/2609.13747/) (2026)

## Abstract (arXiv)

Question Answering (QA) is a longstanding challenge in natural language processing. Existing QA works mostly focus on specific question types, knowledge domains, or reasoning skills. The specialty in QA research hinders systems from modeling commonalities between tasks and generalization for wider applications. To address this issue, we present ProQA, a unified QA paradigm that solves various tasks through a single model. ProQA takes a unified structural prompt as the bridge and improves the QA-centric ability by structural prompt-based pre-training. Through a structurally designed prompt-based input schema, ProQA concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. Furthermore, ProQA is pre-trained with structural prompt-formatted large-scale synthesized corpus, which empowers the model with the commonly-required QA ability. Experimental results on 11 QA benchmarks demonstrate that ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios. Furthermore, ProQA exhibits strong ability in both continual learning and transfer learning by taking the advantages of the structural prompt.