mlsanityData to 14 Oct 2026

Enhancing SLM via ChatGPT and Dataset Augmentation

arXiv 2409.12599 ↗First posted 19 Sep 2024cs.CLcited by 0

On the map: Knowledge distillation → Knowledge distillation for LLMs

Abstract (arXiv)

This paper explores the enhancement of small language models through strategic dataset augmentation via ChatGPT-3.5-Turbo, in the domain of Natural Language Inference (NLI). By employing knowledge distillation-based techniques and synthetic dataset augmentation, we aim to bridge the performance gap between large language models (LLMs) and small language models (SLMs) without the immense cost of human annotation. Our methods involve two forms of rationale generation--information extraction and informed reasoning--to enrich the ANLI dataset. We then fine-tune T5-Small on these augmented datasets, evaluating its performance against an established benchmark. Our findings reveal that the incorporation of synthetic rationales significantly improves the model's ability to comprehend natural language, leading to 1.3% and 2.3% higher classification accuracy, respectively, on the ANLI dataset, demonstrating the potential of leveraging LLMs for dataset augmentation. This approach not only enhances the performance of smaller models on complex tasks but also introduces a cost-effective method for fine-tuning smaller language models. By advancing our understanding of knowledge distillation and fine-tuning strategies, this work contributes to the ongoing effort to create more capable and efficient NLP systems.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 21 of its 27 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

21 of 27 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  7. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  9. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  10. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  11. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  12. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
9 more
  1. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 211
  2. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
  3. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  4. 2020WT5?! Training Text-to-Text Models to Explain their Predictionscited by 148
  5. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  6. 2022FLUTE: Figurative Language Understanding through Textual Explanationscited by 79
  7. 2021When Can Models Learn From Explanations? A Formal Framework for Understanding the Roles of Explanation Datacited by 75
  8. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  9. 2024Show Me How It's Done: The Role of Explanations in Fine-Tuning Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges
  2. 2024Small Language Models: Survey, Measurements, and Insights
  3. 2022Augmenting Interpretable Models with LLMs during Training
  4. 2023LLM-powered Data Augmentation for Enhanced Cross-lingual Performance
  5. 2024A synthetic data approach for domain generalization of NLI models
  6. 2025Can abstract concepts from LLM improve SLM performance?
  7. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  8. 2025MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference