mlsanityData to 7 Oct 2026

The False Promise of Imitating Proprietary LLMs

arXiv 2305.15717 ↗First posted 25 May 2023cs.CLcited by 254

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model, such as a proprietary system like ChatGPT (e.g., Alpaca, Self-Instruct, and others). This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. In this work, we critically analyze this approach. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes (1.5B--13B), data sources, and imitation data amounts (0.3M--150M tokens). We then evaluate the models using crowd raters and canonical NLP benchmarks. Initially, we were surprised by the output quality of our imitation models -- they appear far better at following instructions, and crowd workers rate their outputs as competitive with ChatGPT. However, when conducting more targeted automatic evaluations, we find that imitation models close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data. We show that these performance discrepancies may slip past human raters because imitation models are adept at mimicking ChatGPT's style but not its factuality. Overall, we conclude that model imitation is a false promise: there exists a substantial capabilities gap between open and closed LMs that, with current methods, can only be bridged using an unwieldy amount of imitation data or by using more capable base LMs. In turn, we argue that the highest leverage action for improving open-source models is to tackle the difficult challenge of developing better base LMs, rather than taking the shortcut of imitating proprietary systems.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 30 of its 47 references that are in the map. First posted on arXiv 25 May 2023.

Builds on

30 of 47 references.

References · 30

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
18 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  2. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  4. 2023Instruction Tuning with GPT-4cited by 676
  5. 2016Stealing Machine Learning Models via Prediction APIscited by 602
  6. 2023How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detectioncited by 472
  7. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  8. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  9. 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 285
  10. 2018PRADA: Protecting against DNN Model Stealing Attackscited by 159
  11. 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 136
  12. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
  13. 2019DAWN: Dynamic Adversarial Watermarking of Neural Networkscited by 94
  14. 2019Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attackscited by 79
  15. 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 78
  16. 2019A framework for the extraction of Deep Neural Networks by leveraging public datacited by 22
  17. 2022On the Difficulty of Defending Self-Supervised Learning against Model Extractioncited by 22
  18. 2022Increasing the Cost of Model Extraction with Calibrated Proof of Workcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 88; 2024: 81; 2025: 35; 2026: 50.

254 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  4. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  5. 2023Textbooks Are All You Needcited by 561
  6. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  8. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  9. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  10. 2023Explainability for Large Language Models: A Surveycited by 331
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  12. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
38 more
  1. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  2. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  3. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  4. 2023Challenges and Applications of Large Language Modelscited by 251
  5. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  6. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  7. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 247
  8. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  9. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  10. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  11. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  12. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  13. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  14. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  15. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  16. 2023Self-Alignment with Instruction Backtranslationcited by 149
  17. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
  18. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 126
  19. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  20. 2023Zhongjing: Enhancing the Chinese Medical Capabilities of Large Language Model through Expert Feedback and Real-world Multi-turn Dialoguecited by 118
  21. 2024Language Models Learn to Mislead Humans via RLHFcited by 118
  22. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 112
  23. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 108
  24. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  25. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performancecited by 102
  26. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  27. 2023Frontier AI Regulation: Managing Emerging Risks to Public Safetycited by 95
  28. 2024Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervisioncited by 95
  29. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  30. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  31. 2024Tuning Language Models by Proxycited by 88
  32. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 88
  33. 2024Long Is More for Alignment: A Simple but Tough-to-Beat Baseline for Instruction Fine-Tuningcited by 85
  34. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  35. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systemscited by 84
  36. 2023Training Socially Aligned Language Models on Simulated Social Interactionscited by 77
  37. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  38. 2024A Closer Look at the Limitations of Instruction Tuningcited by 76

50 most cited of 254 All 254 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluations
  3. 2024Your Weak LLM is Secretly a Strong Teacher for Alignment
  4. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  5. 2026Experimental evidence of progressive ChatGPT models self-convergence
  6. 2024Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model
  7. 2026It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs
  8. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets