# Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extraction

arXiv: https://arxiv.org/abs/2303.04132 · HTML: https://mlsanity.com/paper/2303.04132/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 39 references that are in the map.
- First posted on arXiv 7 Mar 2023.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [ZeroGen: Efficient Zero-shot Learning via Dataset Generation](https://mlsanity.com/paper/2202.07922/) (2022)
- [Generating Training Data with Language Models: Towards Zero-Shot Language Understanding](https://mlsanity.com/paper/2202.04538/) (2022)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Not Enough Data? Deep Learning to the Rescue!](https://mlsanity.com/paper/1911.03118/) (2019)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [PEER: A Collaborative Language Model](https://mlsanity.com/paper/2208.11663/) (2022)
- [Towards Zero-Label Language Learning](https://mlsanity.com/paper/2109.09193/) (2021)
- [Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models](https://mlsanity.com/paper/2302.00618/) (2023)
- [GenIE: Generative Information Extraction](https://mlsanity.com/paper/2112.08340/) (2021)
- [Language Models in the Loop: Incorporating Prompting into Weak Supervision](https://mlsanity.com/paper/2205.02318/) (2022)
- [Data Augmentation for Intent Classification with Off-the-shelf Large Language Models](https://mlsanity.com/paper/2204.01959/) (2022)
- [DARE: Data Augmented Relation Extraction with GPT-2](https://mlsanity.com/paper/2004.13845/) (2020)
- [Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions](https://mlsanity.com/paper/2010.10216/) (2020)

## Built on by (most cited first, 30 of 89)

- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey](https://mlsanity.com/paper/2406.15126/) (2024)
- [Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning](https://mlsanity.com/paper/2305.13971/) (2023)
- [Large Language Models for Generative Information Extraction: A Survey](https://mlsanity.com/paper/2312.17617/) (2023)
- [Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!](https://mlsanity.com/paper/2303.08559/) (2023)
- [Large Language Models and Knowledge Graphs: Opportunities and Challenges](https://mlsanity.com/paper/2308.06374/) (2023)
- [Enhancing Large Vision Language Models with Self-Training on Image Comprehension](https://mlsanity.com/paper/2405.19716/) (2024)
- [Self-Exploring Language Models: Active Preference Elicitation for Online Alignment](https://mlsanity.com/paper/2405.19332/) (2024)
- [Agents' Room: Narrative Generation through Multi-step Collaboration](https://mlsanity.com/paper/2410.02603/) (2024)
- [What is the Role of Small Models in the LLM Era: A Survey](https://mlsanity.com/paper/2409.06857/) (2024)
- [Generating Faithful Synthetic Data with Large Language Models: A Case Study in Computational Social Science](https://mlsanity.com/paper/2305.15041/) (2023)
- [People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detection](https://mlsanity.com/paper/2311.01270/) (2023)
- [A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepedia](https://mlsanity.com/paper/2312.02073/) (2023)
- [Evaluating Generative Models for Graph-to-Text Generation](https://mlsanity.com/paper/2307.14712/) (2023)
- [Self-Improving for Zero-Shot Named Entity Recognition with Large Language Models](https://mlsanity.com/paper/2311.08921/) (2023)
- [Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning](https://mlsanity.com/paper/2305.14970/) (2023)
- [Reason from Fallacy: Enhancing Large Language Models' Logical Reasoning through Logical Fallacy Understanding](https://mlsanity.com/paper/2404.04293/) (2024)
- [Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?](https://mlsanity.com/paper/2502.19557/) (2025)
- [Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs](https://mlsanity.com/paper/2402.08005/) (2024)
- [Understanding Synthetic Context Extension via Retrieval Heads](https://mlsanity.com/paper/2410.22316/) (2024)
- [Anyprefer: An Agentic Framework for Preference Data Synthesis](https://mlsanity.com/paper/2504.19276/) (2025)
- [Sketch-Guided Constrained Decoding for Boosting Blackbox Large Language Models without Logit Access](https://mlsanity.com/paper/2401.09967/) (2024)
- [Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models](https://mlsanity.com/paper/2403.00794/) (2024)
- [DiffLM: Controllable Synthetic Data Generation via Diffusion Language Models](https://mlsanity.com/paper/2411.03250/) (2024)
- [The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillation](https://mlsanity.com/paper/2505.18759/) (2025)
- [Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs](https://mlsanity.com/paper/2601.08198/) (2026)
- [Merging Improves Self-Critique Against Jailbreak Attacks](https://mlsanity.com/paper/2406.07188/) (2024)
- [Grounding Data Science Code Generation with Input-Output Specifications](https://mlsanity.com/paper/2402.08073/) (2024)
- [Improving Event Definition Following For Zero-Shot Event Detection](https://mlsanity.com/paper/2403.02586/) (2024)
- [The Effects of Hallucinations in Synthetic Training Data for Relation Extraction](https://mlsanity.com/paper/2410.08393/) (2024)

## Abstract (arXiv)

Large language models (LLMs) have great potential for synthetic data generation. This work shows that useful data can be synthetically generated even for tasks that cannot be solved directly by LLMs: for problems with structured outputs, it is possible to prompt an LLM to perform the task in the reverse direction, by generating plausible input text for a target output structure. Leveraging this asymmetry in task difficulty makes it possible to produce large-scale, high-quality data for complex tasks. We demonstrate the effectiveness of this approach on closed information extraction, where collecting ground-truth data is challenging, and no satisfactory dataset exists to date. We synthetically generate a dataset of 1.8M data points, establish its superior quality compared to existing datasets in a human evaluation, and use it to finetune small models (220M and 770M parameters), termed SynthIE, that outperform the prior state of the art (with equal model size) by a substantial margin of 57 absolute points in micro-F1 and 79 points in macro-F1. Code, data, and models are available at https://github.com/epfl-dlab/SynthIE.