# Crowd-sourcing NLG Data: Pictures Elicit Better Data

arXiv: https://arxiv.org/abs/1608.00339 · HTML: https://mlsanity.com/paper/1608.00339/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2016.

## Built on by (most cited first, 11 of 11)

- [HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation](https://mlsanity.com/paper/2108.06712/) (2021)
- [Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks](https://mlsanity.com/paper/2201.09745/) (2022)
- [Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretability](https://mlsanity.com/paper/2406.18365/) (2024)
- [Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs](https://mlsanity.com/paper/2410.15956/) (2024)
- [Multi-party Goal Tracking with LLMs: Comparing Pre-training, Fine-tuning, and Prompt Engineering](https://mlsanity.com/paper/2308.15231/) (2023)
- [Innovations in Neural Data-to-text Generation: A Survey](https://mlsanity.com/paper/2207.12571/) (2022)
- [Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT](https://mlsanity.com/paper/2109.02938/) (2021)
- [DataTales: A Benchmark for Real-World Intelligent Data Narration](https://mlsanity.com/paper/2410.17859/) (2024)
- [MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages](https://mlsanity.com/paper/2509.26601/) (2025)
- [Self-training from Self-memory in Data-to-text Generation](https://mlsanity.com/paper/2401.10567/) (2024)
- [TEAR: Table Extraction with Attribute Recommendation from Texts via Large Language Models](https://mlsanity.com/paper/2609.15205/) (2026)

## Abstract (arXiv)

Recent advances in corpus-based Natural Language Generation (NLG) hold the promise of being easily portable across domains, but require costly training data, consisting of meaning representations (MRs) paired with Natural Language (NL) utterances. In this work, we propose a novel framework for crowdsourcing high quality NLG training data, using automatic quality control measures and evaluating different MRs with which to elicit data. We show that pictorial MRs result in better NL data being collected than logic-based MRs: utterances elicited by pictorial MRs are judged as significantly more natural, more informative, and better phrased, with a significant increase in average quality ratings (around 0.5 points on a 6-point scale), compared to using the logical MRs. As the MR becomes more complex, the benefits of pictorial stimuli increase. The collected data will be released as part of this submission.