# Controlling Hallucinations at Word Level in Data-to-Text Generation

arXiv: https://arxiv.org/abs/2102.02810 · HTML: https://mlsanity.com/paper/2102.02810/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 77 references that are in the map.
- First posted on arXiv 4 Feb 2021.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Stanza: A Python Natural Language Processing Toolkit for Many Human Languages](https://mlsanity.com/paper/2003.07082/) (2020)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [Toward Controlled Generation of Text](https://mlsanity.com/paper/1703.00955/) (2017)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [The E2E Dataset: New Challenges For End-to-End Generation](https://mlsanity.com/paper/1706.09254/) (2017)
- [OpenNMT: Open-source Toolkit for Neural Machine Translation](https://mlsanity.com/paper/1709.03815/) (2017)
- [Why We Need New Evaluation Metrics for NLG](https://mlsanity.com/paper/1707.06875/) (2017)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [A Persona-Based Neural Conversation Model](https://mlsanity.com/paper/1603.06155/) (2016)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Controlling Linguistic Style Aspects in Neural Language Generation](https://mlsanity.com/paper/1707.02633/) (2017)
- [Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systems](https://mlsanity.com/paper/1508.01745/) (2015)
- [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://mlsanity.com/paper/1906.01081/) (2019)
- [Data-to-Text Generation with Content Selection and Planning](https://mlsanity.com/paper/1809.00582/) (2018)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data](https://mlsanity.com/paper/2010.05873/) (2020)
- [Controlling Output Length in Neural Encoder-Decoders](https://mlsanity.com/paper/1609.09552/) (2016)
- [Semantic Noise Matters for Neural Natural Language Generation](https://mlsanity.com/paper/1911.03905/) (2019)
- [Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generation](https://mlsanity.com/paper/1910.08684/) (2019)
- [Learning Neural Templates for Text Generation](https://mlsanity.com/paper/1808.10122/) (2018)
- [Table-to-text Generation by Structure-aware Seq2seq Learning](https://mlsanity.com/paper/1711.09724/) (2017)

## Built on by (most cited first, 30 of 41)

- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models](https://mlsanity.com/paper/2309.01219/) (2023)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://mlsanity.com/paper/2104.08704/) (2021)
- [Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Code](https://mlsanity.com/paper/2404.00971/) (2024)
- [Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus](https://mlsanity.com/paper/2311.13230/) (2023)
- [Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations](https://mlsanity.com/paper/2503.14477/) (2025)
- [Decoder-Only or Encoder-Decoder? Interpreting Language Model as a Regularized Encoder-Decoder](https://mlsanity.com/paper/2304.04052/) (2023)
- [AI Hallucinations: A Misnomer Worth Clarifying](https://mlsanity.com/paper/2401.06796/) (2024)
- [HallE-Control: Controlling Object Hallucination in Large Multimodal Models](https://mlsanity.com/paper/2310.01779/) (2023)
- [Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods](https://mlsanity.com/paper/2203.05227/) (2022)
- [Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trends](https://mlsanity.com/paper/2409.14457/) (2024)
- [Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation](https://mlsanity.com/paper/2104.07555/) (2021)
- [Neural Pipeline for Zero-Shot Data-to-Text Generation](https://mlsanity.com/paper/2203.16279/) (2022)
- ["Why is this misleading?": Detecting News Headline Hallucinations with Explanations](https://mlsanity.com/paper/2302.05852/) (2023)
- [ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models](https://mlsanity.com/paper/2407.04693/) (2024)
- [The Role of Interactive Visualization in Explaining (Large) NLP Models: from Data to Inference](https://mlsanity.com/paper/2301.04528/) (2023)
- [Hallucination Detection for Grounded Instruction Generation](https://mlsanity.com/paper/2310.15319/) (2023)
- [Critic-Driven Decoding for Mitigating Hallucinations in Data-to-text Generation](https://mlsanity.com/paper/2310.16964/) (2023)
- [ANAH: Analytical Annotation of Hallucinations in Large Language Models](https://mlsanity.com/paper/2405.20315/) (2024)
- [Tackling Hallucinations in Neural Chart Summarization](https://mlsanity.com/paper/2308.00399/) (2023)
- [Sequence-Level Certainty Reduces Hallucination In Knowledge-Grounded Dialogue Generation](https://mlsanity.com/paper/2310.18794/) (2023)
- [SCOPE: A Self-supervised Framework for Improving Faithfulness in Conditional Text Generation](https://mlsanity.com/paper/2502.13674/) (2025)
- [Generating Faithful Text From a Knowledge Graph with Noisy Reference Text](https://mlsanity.com/paper/2308.06488/) (2023)
- [A Systematic Review of Data-to-Text NLG](https://mlsanity.com/paper/2402.08496/) (2024)
- [Steering the Verifiability of Multimodal AI Hallucinations](https://mlsanity.com/paper/2604.06714/) (2026)
- [May the Force Be with Your Copy Mechanism: Enhanced Supervised-Copy Method for Natural Language Generation](https://mlsanity.com/paper/2112.10360/) (2021)
- [Arithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models](https://mlsanity.com/paper/2205.06733/) (2022)
- [Evaluating Parameter Efficient Learning for Generation](https://mlsanity.com/paper/2210.13673/) (2022)
- [Improving Factual Accuracy of Neural Table-to-Text Output by Addressing Input Problems in ToTTo](https://mlsanity.com/paper/2404.04103/) (2024)

## Abstract (arXiv)

Data-to-Text Generation (DTG) is a subfield of Natural Language Generation aiming at transcribing structured data in natural language descriptions. The field has been recently boosted by the use of neural-based generators which exhibit on one side great syntactic skills without the need of hand-crafted pipelines; on the other side, the quality of the generated text reflects the quality of the training data, which in realistic settings only offer imperfectly aligned structure-text pairs. Consequently, state-of-art neural models include misleading statements - usually called hallucinations - in their outputs. The control of this phenomenon is today a major challenge for DTG, and is the problem addressed in the paper. Previous work deal with this issue at the instance level: using an alignment score for each table-reference pair. In contrast, we propose a finer-grained approach, arguing that hallucinations should rather be treated at the word level. Specifically, we propose a Multi-Branch Decoder which is able to leverage word-level labels to learn the relevant parts of each training instance. These labels are obtained following a simple and efficient scoring procedure based on co-occurrence analysis and dependency parsing. Extensive evaluations, via automated metrics and human judgment on the standard WikiBio benchmark, show the accuracy of our alignment labels and the effectiveness of the proposed Multi-Branch Decoder. Our model is able to reduce and control hallucinations, while keeping fluency and coherence in generated texts. Further experiments on a degraded version of ToTTo show that our model could be successfully used on very noisy settings.