# Automatic Dataset Generation for Knowledge Intensive Question Answering Tasks

arXiv: https://arxiv.org/abs/2505.14212 · HTML: https://mlsanity.com/paper/2505.14212/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 43 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [Precise Zero-Shot Dense Retrieval without Relevance Labels](https://mlsanity.com/paper/2212.10496/) (2022)
- [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://mlsanity.com/paper/2403.14403/) (2024)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [Query Rewriting for Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2305.14283/) (2023)
- [RA-DIT: Retrieval-Augmented Dual Instruction Tuning](https://mlsanity.com/paper/2310.01352/) (2023)
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://mlsanity.com/paper/2312.05934/) (2023)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought](https://mlsanity.com/paper/2501.04682/) (2025)
- [Re2G: Retrieve, Rerank, Generate](https://mlsanity.com/paper/2207.06300/) (2022)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [Astute RAG: Overcoming Imperfect Retrieval Augmentation and Knowledge Conflicts for Large Language Models](https://mlsanity.com/paper/2410.07176/) (2024)
- [Dialog Inpainting: Turning Documents into Dialogs](https://mlsanity.com/paper/2205.09073/) (2022)
- [A BERT Baseline for the Natural Questions](https://mlsanity.com/paper/1901.08634/) (2019)
- [The TechQA Dataset](https://mlsanity.com/paper/1911.02984/) (2019)
- [Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Check](https://mlsanity.com/paper/2403.18243/) (2024)

## Built on by (most cited first, 7 of 7)

- [FinStat2SQL: A Text2SQL Pipeline for Financial Statement Analysis](https://mlsanity.com/paper/2506.23273/) (2025)
- [A Benchmark Construction and Evaluation Framework for Specialist Domains: Case Study on Defense-related Documents](https://mlsanity.com/paper/2604.17943/) (2026)
- [D-SCoRE: Document-Centric Segmentation and CoT Reasoning with Structured Export for QA-CoT Data Generation](https://mlsanity.com/paper/2508.01309/) (2025)
- [AmharicStoryQA: A Multicultural Story Question Answering Benchmark in Amharic](https://mlsanity.com/paper/2602.02774/) (2026)
- [Quantifying Hallucinations in Language Language Models on Medical Textbooks](https://mlsanity.com/paper/2603.09986/) (2026)
- [Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA](https://mlsanity.com/paper/2606.32002/) (2026)
- [Investigating the Role of Reasoning-Language Alignment in Monolingual Retrieval-Augmented Generation](https://mlsanity.com/paper/2610.03136/) (2026)

## Abstract (arXiv)

A question-answering (QA) system is to search suitable answers within a knowledge base. Current QA systems struggle with queries requiring complex reasoning or real-time knowledge integration. They are often supplemented with retrieval techniques on a data source such as Retrieval-Augmented Generation (RAG). However, RAG continues to face challenges in handling complex reasoning and logical connections between multiple sources of information. A novel approach for enhancing Large Language Models (LLMs) in knowledge-intensive QA tasks is presented through the automated generation of context-based QA pairs. This methodology leverages LLMs to create fine-tuning data, reducing reliance on human labelling and improving model comprehension and reasoning capabilities. The proposed system includes an automated QA generator and a model fine-tuner, evaluated using perplexity, ROUGE, BLEU, and BERTScore. Comprehensive experiments demonstrate improvements in logical coherence and factual accuracy, with implications for developing adaptable Artificial Intelligence (AI) systems. Mistral-7b-v0.3 outperforms Llama-3-8b with BERT F1, BLEU, and ROUGE scores 0.858, 0.172, and 0.260 of for the LLM generated QA pairs compared to scores of 0.836, 0.083, and 0.139 for the human annotated QA pairs.