# LongForm: Effective Instruction Tuning with Reverse Instructions

arXiv: https://arxiv.org/abs/2304.08460 · HTML: https://mlsanity.com/paper/2304.08460/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 86 references that are in the map.
- First posted on arXiv 17 Apr 2023.

## Builds on (most cited first, 30 of 40)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [No Language Left Behind: Scaling Human-Centered Machine Translation](https://mlsanity.com/paper/2207.04672/) (2022)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [ELI5: Long Form Question Answering](https://mlsanity.com/paper/1907.09190/) (2019)
- [Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus](https://mlsanity.com/paper/2104.08758/) (2021)
- [Texygen: A Benchmarking Platform for Text Generation Models](https://mlsanity.com/paper/1802.01886/) (2018)
- [Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model](https://mlsanity.com/paper/1906.01749/) (2019)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://mlsanity.com/paper/2202.01279/) (2022)
- [Evaluation of Text Generation: A Survey](https://mlsanity.com/paper/2006.14799/) (2020)
- [Explainable Automated Fact-Checking for Public Health Claims](https://mlsanity.com/paper/2010.09926/) (2020)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them](https://mlsanity.com/paper/2102.07033/) (2021)
- [Constituency Parsing with a Self-Attentive Encoder](https://mlsanity.com/paper/1805.01052/) (2018)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [MLSUM: The Multilingual Summarization Corpus](https://mlsanity.com/paper/2004.14900/) (2020)
- [BillSum: A Corpus for Automatic Summarization of US Legislation](https://mlsanity.com/paper/1910.00523/) (2019)

## Built on by (most cited first, 30 of 47)

- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI](https://mlsanity.com/paper/2310.16787/) (2023)
- [SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature](https://mlsanity.com/paper/2406.07835/) (2024)
- [Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation](https://mlsanity.com/paper/2305.14327/) (2023)
- [Weaver: Foundation Models for Creative Writing](https://mlsanity.com/paper/2401.17268/) (2024)
- [Human-Instruction-Free LLM Self-Alignment with Limited Samples](https://mlsanity.com/paper/2401.06785/) (2024)
- [LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm](https://mlsanity.com/paper/2502.19103/) (2025)
- [Investigating Table-to-Text Generation Capabilities of LLMs in Real-World Information Seeking Scenarios](https://mlsanity.com/paper/2305.14987/) (2023)
- [MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions](https://mlsanity.com/paper/2409.12958/) (2024)
- [New Evaluation Metrics Capture Quality Degradation due to LLM Watermarking](https://mlsanity.com/paper/2312.02382/) (2023)
- [Constraint Back-translation Improves Complex Instruction Following of Large Language Models](https://mlsanity.com/paper/2410.24175/) (2024)
- [Semantically-Aware Rewards for Open-Ended R1 Training in Free-Form Generation](https://mlsanity.com/paper/2506.15068/) (2025)
- [LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain](https://mlsanity.com/paper/2404.02127/) (2024)
- [MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://mlsanity.com/paper/2410.23463/) (2024)
- [Enhancing Transformation from Natural Language to Signal Temporal Logic Using LLMs with Diverse External Knowledge](https://mlsanity.com/paper/2505.20658/) (2025)
- [GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation](https://mlsanity.com/paper/2505.20416/) (2025)
- [Aligning Instruction Tuning with Pre-training](https://mlsanity.com/paper/2501.09368/) (2025)
- [Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Models](https://mlsanity.com/paper/2308.12711/) (2023)
- [Hallucination Augmented Recitations for Language Models](https://mlsanity.com/paper/2311.07424/) (2023)
- [SYNTHEVAL: Hybrid Behavioral Testing of NLP Models with Synthetic CheckLists](https://mlsanity.com/paper/2408.17437/) (2024)
- [IterSelectTune: An Iterative Training Framework for Efficient Instruction-Tuning Data Selection](https://mlsanity.com/paper/2410.13464/) (2024)
- [Align to Structure: Aligning Large Language Models with Structural Information](https://mlsanity.com/paper/2504.03622/) (2025)
- [Panza: Design and Analysis of a Fully-Local Personalized Text Writing Assistant](https://mlsanity.com/paper/2407.10994/) (2024)
- [Token-Level Precise Attack on RAG: Searching for the Best Alternatives to Mislead Generation](https://mlsanity.com/paper/2508.03110/) (2025)
- [Diffusion Language Models Are Natively Length-Aware](https://mlsanity.com/paper/2603.06123/) (2026)
- [REInstruct: Building Instruction Data from Unlabeled Corpus](https://mlsanity.com/paper/2408.10663/) (2024)
- [LLM in the Loop: Creating the ParaDeHate Dataset for Hate Speech Detoxification](https://mlsanity.com/paper/2506.01484/) (2025)
- [From General to Targeted Rewards: Surpassing GPT-4 in Open-Ended Long-Context Generation](https://mlsanity.com/paper/2506.16024/) (2025)
- [Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation](https://mlsanity.com/paper/2504.18804/) (2025)
- [Ruler: A Model-Agnostic Method to Control Generated Length for Large Language Models](https://mlsanity.com/paper/2409.18943/) (2024)

## Abstract (arXiv)

Instruction tuning enables language models to more effectively generalize and better follow user intent. However, obtaining instruction data is costly and challenging. Prior work employs methods such as expensive human annotation, crowd-sourced datasets with alignment issues, and generating noisy examples via LLMs. We introduce the LongForm-C dataset, which is created by reverse instructions. We generate instructions via LLMs for human-written corpus examples using reverse instructions. First we select a diverse set of human-written documents from corpora such as C4 and Wikipedia; then we generate instructions for these documents via LLMs. This approach provides a cheaper and cleaner instruction-tuning dataset with natural output and one suitable for long text generation. Our models outperform 10x larger language models without instruction tuning on tasks such as story/recipe generation and long-form question answering. Moreover, LongForm models outperform prior instruction-tuned models such as FLAN-T5 and Alpaca by a large margin, and improve language understanding capabilities further. We publicly release our data and models: https://github.com/akoksal/LongForm.