# Intrinsic Task-based Evaluation for Referring Expression Generation

arXiv: https://arxiv.org/abs/2402.07432 · HTML: https://mlsanity.com/paper/2402.07432/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 26 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Builds on (most cited first, 5 of 5)

- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [NeuralREG: An end-to-end approach to referring expression generation](https://mlsanity.com/paper/1805.08093/) (2018)
- [Non-neural Models Matter: A Re-evaluation of Neural Referring Expression Generation Systems](https://mlsanity.com/paper/2203.08274/) (2022)
- [Referring Expression Generation Using Entity Profiles](https://mlsanity.com/paper/1909.01528/) (2019)
- [Models of reference production: How do they withstand the test of time?](https://mlsanity.com/paper/2307.14817/) (2023)

## Abstract (arXiv)

Recently, a human evaluation study of Referring Expression Generation (REG) models had an unexpected conclusion: on textscwebnlg, Referring Expressions (REs) generated by the state-of-the-art neural models were not only indistinguishable from the REs in textscwebnlg but also from the REs generated by a simple rule-based system. Here, we argue that this limitation could stem from the use of a purely ratings-based human evaluation (which is a common practice in Natural Language Generation). To investigate these issues, we propose an intrinsic task-based evaluation for REG models, in which, in addition to rating the quality of REs, participants were asked to accomplish two meta-level tasks. One of these tasks concerns the referential success of each RE; the other task asks participants to suggest a better alternative for each RE. The outcomes suggest that, in comparison to previous evaluations, the new evaluation protocol assesses the performance of each REG model more comprehensively and makes the participants' ratings more reliable and discriminable.