# On Recipe Memorization and Creativity in Large Language Models: Is Your Model a Creative Cook, a Bad Cook, or Merely a Plagiator?

arXiv: https://arxiv.org/abs/2506.23527 · HTML: https://mlsanity.com/paper/2506.23527/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 21 references that are in the map.
- First posted on arXiv 30 Jun 2025.

## Builds on (most cited first, 9 of 9)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [OLMES: A Standard for Language Model Evaluations](https://mlsanity.com/paper/2406.08446/) (2024)
- [Do Language Models Plagiarize?](https://mlsanity.com/paper/2203.07618/) (2022)
- [RecipeGPT: Generative Pre-training Based Cooking Recipe Generation and Evaluation System](https://mlsanity.com/paper/2003.02498/) (2020)
- [The Authenticity Gap in Human Evaluation](https://mlsanity.com/paper/2205.11930/) (2022)
- ["Mango Mango, How to Let The Lettuce Dry Without A Spinner?": Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner](https://mlsanity.com/paper/2310.05853/) (2023)
- [Monte Carlo Tree Search for Recipe Generation using GPT-2](https://mlsanity.com/paper/2401.05199/) (2024)
- [Large Language Models as Sous Chefs: Revising Recipes with GPT-3](https://mlsanity.com/paper/2306.13986/) (2023)

## Abstract (arXiv)

This work-in-progress investigates the memorization, creativity, and nonsense found in cooking recipes generated from Large Language Models (LLMs). Precisely, we aim (i) to analyze memorization, creativity, and non-sense in LLMs using a small, high-quality set of human judgments and (ii) to evaluate potential approaches to automate such a human annotation in order to scale our study to hundreds of recipes. To achieve (i), we conduct a detailed human annotation on 20 preselected recipes generated by LLM (Mixtral), extracting each recipe's ingredients and step-by-step actions to assess which elements are memorized--i.e., directly traceable to online sources possibly seen during training--and which arise from genuine creative synthesis or outright nonsense. We find that Mixtral consistently reuses ingredients that can be found in online documents, potentially seen during model training, suggesting strong reliance on memorized content. To achieve aim (ii) and scale our analysis beyond small sample sizes and single LLM validation, we design an ``LLM-as-judge'' pipeline that automates recipe generation, nonsense detection, parsing ingredients and recipe steps, and their annotation. For instance, comparing its output against human annotations, the best ingredient extractor and annotator is Llama 3.1+Gemma 2 9B, achieving up to 78% accuracy on ingredient matching. This automated framework enables large-scale quantification of memorization, creativity, and nonsense in generated recipes, providing rigorous evidence of the models' creative capacities.