# The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation

arXiv: https://arxiv.org/abs/2311.05552 · HTML: https://mlsanity.com/paper/2311.05552/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 90 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [Content Planning for Neural Story Generation with Aristotelian Rescoring](https://mlsanity.com/paper/2009.09870/) (2020)
- [Iteratively Prompt Pre-trained Language Models for Chain of Thought](https://mlsanity.com/paper/2203.08383/) (2022)
- [Making Science Simple: Corpora for the Lay Summarisation of Scientific Literature](https://mlsanity.com/paper/2210.09932/) (2022)
- [Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing](https://mlsanity.com/paper/2104.10097/) (2021)
- [Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions](https://mlsanity.com/paper/2205.00415/) (2022)
- [Pun Generation with Surprise](https://mlsanity.com/paper/1904.06828/) (2019)
- [A Report on the 2020 Sarcasm Detection Shared Task](https://mlsanity.com/paper/2005.05814/) (2020)
- [AmbiPun: Generating Humorous Puns with Ambiguous Context](https://mlsanity.com/paper/2205.01825/) (2022)
- [ExPUNations: Augmenting Puns with Keywords and Explanations](https://mlsanity.com/paper/2210.13513/) (2022)
- [Pun-GAN: Generative Adversarial Network for Pun Generation](https://mlsanity.com/paper/1910.10950/) (2019)
- [Alexa, Google, Siri: What are Your Pronouns? Gender and Anthropomorphism in the Design and Perception of Conversational Assistants](https://mlsanity.com/paper/2106.02578/) (2021)
- [TwistList: Resources and Baselines for Tongue Twister Generation](https://mlsanity.com/paper/2306.03457/) (2023)
- [SemEval-2020 Task 7: Assessing Humor in Edited News Headlines](https://mlsanity.com/paper/2008.00304/) (2020)
- [A Survey of NLP-Related Crowdsourcing HITs: what works and what does not](https://mlsanity.com/paper/2111.05241/) (2021)
- [NGEP: A Graph-based Event Planning Framework for Story Generation](https://mlsanity.com/paper/2210.10602/) (2022)
- [A Unified Framework for Pun Generation with Humor Principles](https://mlsanity.com/paper/2210.13055/) (2022)
- ["Judge me by my size (noun), do you?'' YodaLib: A Demographic-Aware Humor Generation Framework](https://mlsanity.com/paper/2006.00578/) (2020)
- [Robots-Dont-Cry: Understanding Falsely Anthropomorphic Utterances in Dialog Systems](https://mlsanity.com/paper/2210.12429/) (2022)
- [Pick a Fight or Bite your Tongue: Investigation of Gender Differences in Idiomatic Language Usage](https://mlsanity.com/paper/2011.00335/) (2020)
- [A Survey in Automatic Irony Processing: Linguistic, Cognitive, and Multi-X Perspectives](https://mlsanity.com/paper/2209.04712/) (2022)
- [Assessing Digital Language Support on a Global Scale](https://mlsanity.com/paper/2209.13515/) (2022)
- [Punny Captions: Witty Wordplay in Image Descriptions](https://mlsanity.com/paper/1704.08224/) (2017)
- [Conservative AI and social inequality: Conceptualizing alternatives to bias through social theory](https://mlsanity.com/paper/2007.08666/) (2020)
- [Implicit Gender Bias in Computer Science -- A Qualitative Study](https://mlsanity.com/paper/2107.01624/) (2021)
- ["When Words Fail, Emojis Prevail": Generating Sarcastic Utterances with Emoji Using Valence Reversal and Semantic Incongruity](https://mlsanity.com/paper/2305.04105/) (2023)

## Built on by (most cited first, 11 of 11)

- [Position: Towards Bidirectional Human-AI Alignment](https://mlsanity.com/paper/2406.09264/) (2024)
- [SLIDE: A Framework Integrating Small and Large Language Models for Open-Domain Dialogues Evaluation](https://mlsanity.com/paper/2405.15924/) (2024)
- [MMTE: Corpus and Metrics for Evaluating Machine Translation Quality of Metaphorical Language](https://mlsanity.com/paper/2406.13698/) (2024)
- [Humor Mechanics: Advancing Humor Generation with Multistep Reasoning](https://mlsanity.com/paper/2405.07280/) (2024)
- [Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes](https://mlsanity.com/paper/2507.13335/) (2025)
- [Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts](https://mlsanity.com/paper/2504.21117/) (2025)
- [Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depth](https://mlsanity.com/paper/2509.03867/) (2025)
- [Train & Constrain: Phonologically Informed Tongue-Twister Generation from Topics and Paraphrases](https://mlsanity.com/paper/2403.13901/) (2024)
- [DRE: An Effective Dual-Refined Method for Integrating Small and Large Language Models in Open-Domain Dialogue Evaluation](https://mlsanity.com/paper/2506.04516/) (2025)
- [ReproHum #0087-01: Human Evaluation Reproduction Report for Generating Fact Checking Explanations](https://mlsanity.com/paper/2404.17481/) (2024)
- [Natural Language Generation](https://mlsanity.com/paper/2503.16728/) (2025)

## Abstract (arXiv)

Human evaluation is often considered to be the gold standard method of evaluating a Natural Language Generation system. However, whilst its importance is accepted by the community at large, the quality of its execution is often brought into question. In this position paper, we argue that the generation of more esoteric forms of language - humour, irony and sarcasm - constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance, and every effort should be made to report demographic characteristics wherever possible, in the interest of transparency and replicability. We support these claims with an overview of each language form and an analysis of examples in terms of how their interpretation is affected by different participant variables. We additionally perform a critical survey of recent works in NLG to assess how well evaluation procedures are reported in this subdomain, and note a severe lack of open reporting of evaluator demographic information, and a significant reliance on crowdsourcing platforms for recruitment.