mlsanityData to 7 Oct 2026

Symbolic Knowledge Distillation: from General Language Models to Commonsense Models

arXiv 2110.07178 ↗First posted 14 Oct 2021cs.CLcited by 270

Not on the map. Why?

Cited by 43 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The common practice for training commonsense models has gone from-human-to-corpus-to-machine: humans author commonsense knowledge graphs in order to train commonsense models. In this work, we investigate an alternative, from-machine-to-corpus-to-machine: general language models author these commonsense knowledge graphs to train commonsense models. Our study leads to a new framework, Symbolic Knowledge Distillation. As with prior art in Knowledge Distillation (Hinton et al., 2015), our approach uses larger models to teach smaller models. A key difference is that we distill knowledge symbolically-as text-in addition to the neural model. We also distill only one aspect-the commonsense of a general language model teacher, allowing the student to be a different type, a commonsense model. Altogether, we show that careful prompt engineering and a separately trained critic model allow us to selectively distill high-quality causal commonsense from GPT-3, a general language model. Empirical results demonstrate that, for the first time, a human-authored commonsense knowledge graph is surpassed by our automatically distilled variant in all three criteria: quantity, quality, and diversity. In addition, it results in a neural commonsense model that surpasses the teacher model's commonsense capabilities despite its 100x smaller size. We apply this to the ATOMIC resource, and share our new symbolic knowledge graph and commonsense models.

Key facts. Cited by 270 papers in the map (data through 7 Oct 2026). Builds on 41 of its 76 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

41 of 76 references.

References · 41

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
29 more
  1. 2016Sequence-Level Knowledge Distillationcited by 761
  2. 2018Annotation Artifacts in Natural Language Inference Datacited by 656
  3. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  4. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  5. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
  6. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  7. 2020COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphscited by 254
  8. 2021Generating Datasets with Pretrained Language Modelscited by 192
  9. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  10. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  11. 2019Are We Modeling the Task or the Annotator? An Investigation of Annotator Bias in Natural Language Understanding Datasetscited by 173
  12. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  13. 2020Adversarial Filters of Dataset Biasescited by 159
  14. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  15. 2019ASER: A Large-scale Eventuality Knowledge Graphcited by 89
  16. 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 61
  17. 2018Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailmentcited by 58
  18. 2021Provable Limitations of Acquiring Meaning from Ungrounded Form: What Will Future Language Models Understand?cited by 55
  19. 2020TransOMCS: From Linguistic Graphs to Commonsense Knowledgecited by 52
  20. 2020GenericsKB: A Knowledge Base of Generic Statementscited by 51
  21. 2021MERMAID: Metaphor Generation with Symbolism and Discriminative Decodingcited by 49
  22. 2020DARE: Data Augmented Relation Extraction with GPT-2cited by 43
  23. 2017The Effect of Different Writing Tasks on Linguistic Style: A Case Study of the ROC Story Cloze Taskcited by 39
  24. 2020Temporal Common Sense Acquisition with Minimal Supervisioncited by 38
  25. 2020Automated Storytelling via Causal, Commonsense Plot Orderingcited by 38
  26. 2020How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worldscited by 35
  27. 2021Guided Generation of Cause and Effectcited by 33
  28. 2020R3: Reverse, Retrieve, and Rank for Sarcasm Generation with Commonsense Knowledgecited by 32
  29. 2021Conversational Multi-Hop Reasoning with Neural Commonsense Knowledge and Symbolic Logic Rulescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 44; 2023: 89; 2024: 80; 2025: 31; 2026: 20.

270 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  4. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  5. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 663
  6. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  7. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  8. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 350
  9. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  10. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  11. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  12. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
38 more
  1. 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 235
  2. 2022Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMscited by 220
  3. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  4. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  5. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  6. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  8. 2022A Review on Language Models as Knowledge Basescited by 156
  9. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  10. 2021KAT: A Knowledge Augmented Transformer for Vision-and-Languagecited by 125
  11. 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
  12. 2022ProsocialDialog: A Prosocial Backbone for Conversational Agentscited by 109
  13. 2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 106
  14. 2023LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunitiescited by 97
  15. 2023Value Kaleidoscope: Engaging AI with Pluralistic Human Values, Rights, and Dutiescited by 97
  16. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  17. 2023When Large Language Models Meet Personalization: Perspectives of Challenges and Opportunitiescited by 89
  18. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  19. 2021Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefscited by 79
  20. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  21. 2022Extracting Cultural Commonsense Knowledge at Scalecited by 76
  22. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  23. 2023Agent Lumos: Unified and Modular Training for Open-Source Language Agentscited by 72
  24. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  25. 2023ChatGPT is a Knowledgeable but Inexperienced Solver: An Investigation of Commonsense Problem in Large Language Modelscited by 63
  26. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  27. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  28. 2022Revisiting DocRED -- Addressing the False Negative Problem in Relation Extractioncited by 57
  29. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 56
  30. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  31. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  32. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 55
  33. 2024Understanding the Capabilities and Limitations of Large Language Models for Cultural Commonsensecited by 50
  34. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  35. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  36. 2023Distilling Script Knowledge from Large Language Models for Constrained Language Planningcited by 45
  37. 2023Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statementscited by 44
  38. 2024A Survey on Transformer Compressioncited by 44

50 most cited of 270 All 270 (JSON)