mlsanityData to 14 Oct 2026

TempLM: Distilling Language Models into Template-Based Generators

arXiv 2205.11055 ↗First posted 23 May 2022cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

While pretrained language models (PLMs) have greatly improved text generation, they have also been known to produce unfaithful or inappropriate content. In contrast, classic template-based systems provide strong guarantees of faithfulness at the cost of fluency. We propose TempLM, which achieves the best of both worlds by distilling a PLM into a template-based generator. On the E2E and SynthBio data-to-text datasets, we show that TempLM is more faithful than the original PLM and is more fluent than prior template systems. Notably, on an out-of-domain evaluation, TempLM reduces a finetuned BART model's unfaithfulness rate from 83% to 0%. In a human study, we find that TempLM's templates substantially improve upon human-written ones in BERTScore.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 21 of its 39 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

21 of 39 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2020Extracting Training Data from Large Language Modelscited by 2,189
  7. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  9. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  10. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  11. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  12. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 253
9 more
  1. 2018Constituency Parsing with a Self-Attentive Encodercited by 199
  2. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  3. 2016Data Recombination for Neural Semantic Parsingcited by 111
  4. 2018Data-to-Text Generation with Content Selection and Planningcited by 106
  5. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  6. 2018Learning Neural Templates for Text Generationcited by 61
  7. 2020Variational Template Machine for Data-to-Text Generationcited by 19
  8. 2020Posterior Control of Blackbox Generationcited by 16
  9. 2021Data-to-text Generation by Splicing Together Nearest Neighborscited by 4

Built on by

Cited by year

Citing papers per year: 2023: 4.

4 papers.

  1. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  2. 2023Efficient Large Language Models: A Surveycited by 186
  3. 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 34
  4. 2023Towards Verifiable Text Generation with Symbolic Referencescited by 13