# Does Synthetic Data Make Large Language Models More Efficient?

arXiv: https://arxiv.org/abs/2310.07830 · HTML: https://mlsanity.com/paper/2310.07830/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 11 Oct 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Improving Neural Machine Translation Models with Monolingual Data](https://mlsanity.com/paper/1511.06709/) (2015)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [Training Question Answering Models From Synthetic Data](https://mlsanity.com/paper/2002.09599/) (2020)
- [A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task](https://mlsanity.com/paper/1606.02858/) (2016)
- [Improving Neural Question Generation using Answer Separation](https://mlsanity.com/paper/1809.02393/) (2018)
- [Do Generative Large Language Models need billions of parameters?](https://mlsanity.com/paper/2309.06589/) (2023)
- [Zero-Shot Open-Book Question Answering](https://mlsanity.com/paper/2111.11520/) (2021)
- [Can pruning make Large Language Models more efficient?](https://mlsanity.com/paper/2310.04573/) (2023)
- [Alexa, Predict My Flight Delay](https://mlsanity.com/paper/2208.09921/) (2022)
- [Can a student Large Language Model perform as well as it's teacher?](https://mlsanity.com/paper/2310.02421/) (2023)

## Built on by (most cited first, 8 of 8)

- [A Case Study Exploring the Current Landscape of Synthetic Medical Record Generation with Commercial LLMs](https://mlsanity.com/paper/2504.14657/) (2025)
- [RouteNator: A Router-Based Multi-Modal Architecture for Generating Synthetic Training Data for Function Calling LLMs](https://mlsanity.com/paper/2505.10495/) (2025)
- [SkillScope: A Tool to Predict Fine-Grained Skills Needed to Solve Issues on GitHub](https://mlsanity.com/paper/2501.15922/) (2025)
- [NeSy is alive and well: A LLM-driven symbolic approach for better code comment data generation and classification](https://mlsanity.com/paper/2402.16910/) (2024)
- [Backbone Augmented Training for Adaptations](https://mlsanity.com/paper/2506.04288/) (2025)
- [The Synthetic Mirror -- Synthetic Data at the Age of Agentic AI](https://mlsanity.com/paper/2506.13818/) (2025)
- [Sub-Billion, Super-Frontier: Small Language Models Rival Zero-Shot Frontier LLMs on General and Literary Relation Extraction](https://mlsanity.com/paper/2606.22606/) (2026)
- [Incremental Risk Assessment of Progressive Elder Financial Scams via Instruction-Tuned Small Language Models](https://mlsanity.com/paper/2609.00005/) (2026)

## Abstract (arXiv)

Natural Language Processing (NLP) has undergone transformative changes with the advent of deep learning methodologies. One challenge persistently confronting researchers is the scarcity of high-quality, annotated datasets that drive these models. This paper explores the nuances of synthetic data generation in NLP, with a focal point on template-based question generation. By assessing its advantages, including data augmentation potential and the introduction of structured variety, we juxtapose these benefits against inherent limitations, such as the risk of overfitting and the constraints posed by pre-defined templates. Drawing from empirical evaluations, we demonstrate the impact of template-based synthetic data on the performance of modern transformer models. We conclude by emphasizing the delicate balance required between synthetic and real-world data, and the future trajectories of integrating synthetic data in model training pipelines. The findings aim to guide NLP practitioners in harnessing synthetic data's potential, ensuring optimal model performance in diverse applications.