# Semantic Noise Matters for Neural Natural Language Generation

arXiv: https://arxiv.org/abs/1911.03905 · HTML: https://mlsanity.com/paper/1911.03905/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Nov 2019.

## Built on by (most cited first, 30 of 65)

- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics](https://mlsanity.com/paper/2102.01672/) (2021)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning](https://mlsanity.com/paper/2111.10952/) (2021)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [Unified Demonstration Retriever for In-Context Learning](https://mlsanity.com/paper/2305.04320/) (2023)
- [CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP](https://mlsanity.com/paper/2104.08835/) (2021)
- [Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy](https://mlsanity.com/paper/2210.17546/) (2022)
- [Representation Noising: A Defence Mechanism Against Harmful Finetuning](https://mlsanity.com/paper/2405.14577/) (2024)
- [UPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation](https://mlsanity.com/paper/2303.08518/) (2023)
- [In-context Learning with Retrieved Demonstrations for Language Models: A Survey](https://mlsanity.com/paper/2401.11624/) (2024)
- [Do Androids Know They're Only Dreaming of Electric Sheep?](https://mlsanity.com/paper/2312.17249/) (2023)
- [A Multi-Task Embedder For Retrieval Augmented LLMs](https://mlsanity.com/paper/2310.07554/) (2023)
- [Learning to Retrieve In-Context Examples for Large Language Models](https://mlsanity.com/paper/2307.07164/) (2023)
- [AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model](https://mlsanity.com/paper/2208.01448/) (2022)
- [Adapting Large Language Models to Domains via Reading Comprehension](https://mlsanity.com/paper/2309.09530/) (2023)
- [Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder](https://mlsanity.com/paper/2304.04052/) (2023)
- [Controlling Hallucinations at Word Level in Data-to-Text Generation](https://mlsanity.com/paper/2102.02810/) (2021)
- [GEMv2: Multilingual NLG Benchmarking in a Single Line of Code](https://mlsanity.com/paper/2206.11249/) (2022)
- [AI Hallucinations: A Misnomer Worth Clarifying](https://mlsanity.com/paper/2401.06796/) (2024)
- [Learning to Revise References for Faithful Summarization](https://mlsanity.com/paper/2204.10290/) (2022)
- [Exploring Universal Intrinsic Task Subspace via Prompt Tuning](https://mlsanity.com/paper/2110.07867/) (2021)
- [Structural Information Preserving for Graph-to-Text Generation](https://mlsanity.com/paper/2102.06749/) (2021)
- [Control Prefixes for Parameter-Efficient Text Generation](https://mlsanity.com/paper/2110.08329/) (2021)
- [Underreporting of errors in NLG output, and what to do about it](https://mlsanity.com/paper/2108.01182/) (2021)
- [Neural Pipeline for Zero-Shot Data-to-Text Generation](https://mlsanity.com/paper/2203.16279/) (2022)

## Abstract (arXiv)

Neural natural language generation (NNLG) systems are known for their pathological outputs, i.e. generating text which is unrelated to the input specification. In this paper, we show the impact of semantic noise on state-of-the-art NNLG models which implement different semantic control mechanisms. We find that cleaned data can improve semantic correctness by up to 97%, while maintaining fluency. We also find that the most common error is omitting information, rather than hallucination.