mlsanityData to 14 Oct 2026

Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing

arXiv 2305.16635 ↗First posted 26 May 2023cs.CL cs.AI cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

We present Impossible Distillation, a novel framework for paraphrasing and sentence summarization, that distills a high-quality dataset and model from a low-quality teacher that itself cannot perform these tasks. Unlike prior works that rely on an extreme-scale teacher model (e.g., GPT3) or task-specific architecture, we hypothesize and verify the paraphrastic proximity intrinsic to pre-trained LMs (e.g., GPT2), where paraphrases occupy a proximal subspace in the LM distribution. By identifying and distilling generations from these subspaces, Impossible Distillation produces a high-quality dataset and model even from GPT2-scale LMs. We evaluate our method on multiple benchmarks spanning unconstrained / syntax-controlled paraphrase generation and sentence summarization. Our model with 770M parameters consistently outperforms strong baselines, including models distilled from ChatGPT, and sometimes, even ChatGPT itself. Also, we find that our distilled dataset from 1.5B LMs exhibits higher diversity and fidelity than up to 13 times larger datasets.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 45 of its 70 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

45 of 70 references.

References · 45

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  10. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
33 more
  1. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  4. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  5. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  6. 2015A Neural Attention Model for Abstractive Sentence Summarizationcited by 396
  7. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 297
  8. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  9. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  10. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  11. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  12. 2021Data Augmentation Can Improve Robustnesscited by 194
  13. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 168
  14. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  15. 2019Lagging Inference Networks and Posterior Collapse in Variational Autoencoderscited by 124
  16. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 94
  17. 2020Syntax-guided Controlled Generation of Paraphrasescited by 47
  18. 2021Can NLI Models Verify QA Systems' Predictions?cited by 46
  19. 2021Generating Syntactically Controlled Paraphrases without Using Annotated Parallel Pairscited by 45
  20. 2019Controllable Paraphrase Generation with a Syntactic Exemplarcited by 44
  21. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  22. 2019ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translationcited by 38
  23. 2022How Large Language Models are Transforming Machine-Paraphrased Plagiarismcited by 38
  24. 2022Prompting as Probing: Using Language Models for Knowledge Base Constructioncited by 32
  25. 2021ParaSCI: A Large Scientific Paraphrase Dataset for Longer Paraphrase Generationcited by 28
  26. 2019SEQ^3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compressioncited by 27
  27. 2023GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilitiescited by 22
  28. 2022I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitationcited by 21
  29. 2019Simple Unsupervised Summarization by Contextual Matchingcited by 18
  30. 2018Unsupervised Sentence Compression using Denoising Auto-Encoderscited by 15
  31. 2022Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillationcited by 14
  32. 2021ConRPG: Paraphrase Generation using Contexts as Regularizercited by 13
  33. 2023InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPTcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 15; 2024: 9; 2025: 5; 2026: 2.

31 papers.

Sort:

By area:

  1. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  2. 2023Textbooks Are All You Needcited by 569
  3. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 284
  5. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  6. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  7. 2023UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognitioncited by 72
  8. 2023Large Language Models to Identify Social Determinants of Health in Electronic Health Recordscited by 55
  9. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 41
  10. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  11. 2023LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compressioncited by 26
  12. 2023Teaching Language Models to Self-Improve through Interactive Demonstrationscited by 26
19 more
  1. 2023UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situationscited by 19
  2. 2023What Makes it Ok to Set a Fire? Iterative Self-distillation of Contexts and Rationales for Disambiguating Defeasible Social and Moral Situationscited by 15
  3. 2023STEER: Unified Style Transfer with Expert Reinforcementcited by 14
  4. 2024SynthesizRR: Generating Diverse Datasets with Retrieval Augmentationcited by 13
  5. 2023Fine-Tuning Language Models Using Formal Methods Feedbackcited by 12
  6. 2025Aligning Large Language Models to Follow Instructions and Hallucinate Less via Effective Data Filteringcited by 12
  7. 2023Chain-of-Thought Reasoning is a Policy Improvement Operatorcited by 8
  8. 2023NovaCOMET: Open Commonsense Foundation Models with Symbolic Knowledge Distillationcited by 8
  9. 2024JAMDEC: Unsupervised Authorship Obfuscation using Constrained Decoding over Small Language Modelscited by 7
  10. 2025Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learningcited by 7
  11. 2024Information-Theoretic Distillation for Reference-less Summarizationcited by 6
  12. 2024DivTOD: Unleashing the Power of LLMs for Diversifying Task-Oriented Dialogue Representationscited by 4
  13. 2024MMIDR: Teaching Large Language Model to Interpret Multimodal Misinformation via Knowledge Distillationcited by 3
  14. 2024Training Task Experts through Retrieval Based Distillationcited by 3
  15. 2025KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillationcited by 3
  16. 2025Stratos: An End-to-End Distillation Pipeline for Customized LLMs under Distributed Cloud Environmentscited by 1
  17. 2023From Lengthy to Lucid: A Systematic Literature Review on NLP Techniques for Taming Long Sentencescited by 0
  18. 2026Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMscited by 0
  19. 2026Cross-Domain, Multi-Task Data-to-Text Generation without In-Domain Training Datacited by 0