# Genie: Achieving Human Parity in Content-Grounded Datasets Generation

arXiv: https://arxiv.org/abs/2401.14367 · HTML: https://mlsanity.com/paper/2401.14367/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 25 Jan 2024.

## Builds on (most cited first, 30 of 31)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [ASQA: Factoid Questions Meet Long-Form Answers](https://mlsanity.com/paper/2204.06092/) (2022)
- [SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization](https://mlsanity.com/paper/2212.10465/) (2022)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answering](https://mlsanity.com/paper/2307.16877/) (2023)
- [Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering](https://mlsanity.com/paper/2104.08202/) (2021)
- [DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering](https://mlsanity.com/paper/2211.05655/) (2022)
- [Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://mlsanity.com/paper/2105.00071/) (2021)
- [Efficient Benchmarking of Language Models](https://mlsanity.com/paper/2308.11696/) (2023)
- [Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge](https://mlsanity.com/paper/2305.01651/) (2023)
- [Dialog Inpainting: Turning Documents into Dialogs](https://mlsanity.com/paper/2205.09073/) (2022)
- [Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand](https://mlsanity.com/paper/2112.04139/) (2021)
- [Automatically Extracting Challenge Sets for Non local Phenomena in Neural Machine Translation](https://mlsanity.com/paper/1909.06814/) (2019)

## Built on by (most cited first, 30 of 31)

- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [Instruction Pre-Training: Language Models are Supervised Multitask Learners](https://mlsanity.com/paper/2406.14491/) (2024)
- [Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation](https://mlsanity.com/paper/2402.18334/) (2024)
- [I-SHEEP: Self-Alignment of LLM from Scratch through an Iterative Self-Enhancement Paradigm](https://mlsanity.com/paper/2408.08072/) (2024)
- [AmbigDocs: Reasoning across Documents on Different Entities under the Same Name](https://mlsanity.com/paper/2404.12447/) (2024)
- [CodeUpdateArena: Benchmarking Knowledge Editing on API Updates](https://mlsanity.com/paper/2407.06249/) (2024)
- [Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models](https://mlsanity.com/paper/2412.02674/) (2024)
- [Code Needs Comments: Enhancing Code LLMs with Comment Augmentation](https://mlsanity.com/paper/2402.13013/) (2024)
- [Naturally Occurring Feedback is Common, Extractable and Useful](https://mlsanity.com/paper/2407.10944/) (2024)
- [Advancing Large Language Model Attribution through Self-Improving](https://mlsanity.com/paper/2410.13298/) (2024)
- [SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation](https://mlsanity.com/paper/2405.10040/) (2024)
- [Importance Weighting Can Help Large Language Models Self-Improve](https://mlsanity.com/paper/2408.09849/) (2024)
- [ROC-n-reroll: How verifier imperfection affects test-time scaling](https://mlsanity.com/paper/2507.12399/) (2025)
- [LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexicons](https://mlsanity.com/paper/2402.14086/) (2024)
- [FANNO: Augmenting High-Quality Instruction Data with Open-Sourced LLMs Only](https://mlsanity.com/paper/2408.01323/) (2024)
- [What Has Been Lost with Synthetic Evaluation?](https://mlsanity.com/paper/2505.22830/) (2025)
- [GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation](https://mlsanity.com/paper/2505.20416/) (2025)
- [LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuning](https://mlsanity.com/paper/2502.14644/) (2025)
- [LLM Distillation for Efficient Few-Shot Multiple Choice Question Answering](https://mlsanity.com/paper/2412.09807/) (2024)
- [PPFedIT: Towards Privacy-Preserving Federated Instruction Tuning with Few-shot Local Examples](https://mlsanity.com/paper/2403.06131/) (2024)
- [CorrSynth -- A Correlated Sampling Method for Diverse Dataset Generation from LLMs](https://mlsanity.com/paper/2411.08553/) (2024)
- [CLEAR: Error Analysis via LLM-as-a-Judge Made Easy](https://mlsanity.com/paper/2507.18392/) (2025)
- [An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation](https://mlsanity.com/paper/2505.03452/) (2025)
- [Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression](https://mlsanity.com/paper/2406.11354/) (2024)
- [Leveraging Synthetic Data for Question Answering with Multilingual LLMs in the Agricultural Domain](https://mlsanity.com/paper/2507.16974/) (2025)
- [Retrieving Contextual Information for Long-Form Question Answering using Weak Supervision](https://mlsanity.com/paper/2410.08623/) (2024)
- [Hypothesis-only Biases in Large Language Model-Elicited Natural Language Inference](https://mlsanity.com/paper/2410.08996/) (2024)
- [Beyond Static Benchmarks: Synthesizing Harmful Content via Persona-based Simulation for Robust Evaluation](https://mlsanity.com/paper/2604.17020/) (2026)
- [Knowledge-to-Verification: Exploring RLVR for LLMs in Knowledge-Intensive Domains](https://mlsanity.com/paper/2605.18261/) (2026)
- [Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA](https://mlsanity.com/paper/2606.32002/) (2026)

## Abstract (arXiv)

The lack of high-quality data for content-grounded generation tasks has been identified as a major obstacle to advancing these tasks. To address this gap, we propose Genie, a novel method for automatically generating high-quality content-grounded data. It consists of three stages: (a) Content Preparation, (b) Generation: creating task-specific examples from the content (e.g., question-answer pairs or summaries). (c) Filtering mechanism aiming to ensure the quality and faithfulness of the generated data. We showcase this methodology by generating three large-scale synthetic data, making wishes, for Long-Form Question-Answering (LFQA), summarization, and information extraction. In a human evaluation, our generated data was found to be natural and of high quality. Furthermore, we compare models trained on our data with models trained on human-written data -- ELI5 and ASQA for LFQA and CNN-DailyMail for Summarization. We show that our models are on par with or outperforming models trained on human-generated data and consistently outperforming them in faithfulness. Finally, we applied our method to create LFQA data within the medical domain and compared a model trained on it with models trained on other domains.