# GENIE: A Fine-Grained Measure for Novelty

arXiv: https://arxiv.org/abs/2606.12790 · HTML: https://mlsanity.com/paper/2606.12790/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Builds on (most cited first, 30 of 30)

- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [Does Writing with Language Models Reduce Content Diversity?](https://mlsanity.com/paper/2309.05196/) (2023)
- [Art or Artifice? Large Language Models and the False Promise of Creativity](https://mlsanity.com/paper/2309.14556/) (2023)
- [Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)](https://mlsanity.com/paper/2510.22954/) (2025)
- [Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity](https://mlsanity.com/paper/2510.01171/) (2025)
- [Agents' Room: Narrative Generation through Multi-step Collaboration](https://mlsanity.com/paper/2410.02603/) (2024)
- [Base Models Beat Aligned Models at Randomness and Creativity](https://mlsanity.com/paper/2505.00047/) (2025)
- [Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Edits](https://mlsanity.com/paper/2409.14509/) (2024)
- [Echoes in AI: Quantifying lack of plot diversity in LLM outputs](https://mlsanity.com/paper/2501.00273/) (2024)
- [Detection and Measurement of Syntactic Templates in Generated Text](https://mlsanity.com/paper/2407.00211/) (2024)
- [We're Different, We're the Same: Creative Homogeneity Across LLMs](https://mlsanity.com/paper/2501.19361/) (2025)
- [LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing](https://mlsanity.com/paper/2507.00769/) (2025)
- [Exploring Precision and Recall to assess the quality and diversity of LLMs](https://mlsanity.com/paper/2402.10693/) (2024)
- [Evaluating n-Gram Novelty of Language Models Using Rusty-DAWG](https://mlsanity.com/paper/2406.13069/) (2024)
- [AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computation](https://mlsanity.com/paper/2504.07532/) (2025)
- [Creative Preference Optimization](https://mlsanity.com/paper/2505.14442/) (2025)
- [Diverse Preference Learning for Capabilities and Alignment](https://mlsanity.com/paper/2511.08594/) (2025)
- [Detecting Mode Collapse in Language Models via Narration](https://mlsanity.com/paper/2402.04477/) (2024)
- [QUDsim: Quantifying Discourse Similarities in LLM-Generated Text](https://mlsanity.com/paper/2504.09373/) (2025)
- [NarraBench: A Comprehensive Framework for Narrative Benchmarking](https://mlsanity.com/paper/2510.09869/) (2025)
- [Evaluating Text Creativity across Diverse Domains: A Dataset and Large Language Model Evaluator](https://mlsanity.com/paper/2505.19236/) (2025)
- [StoryScope: Investigating idiosyncrasies in AI fiction](https://mlsanity.com/paper/2604.03136/) (2026)
- [LLMs Corrupt Your Documents When You Delegate](https://mlsanity.com/paper/2604.15597/) (2026)
- [Frankentext: Stitching random text fragments into long-form narratives](https://mlsanity.com/paper/2505.18128/) (2025)
- [Measuring Combinatorial Creativity in Large Language Models and Humans](https://mlsanity.com/paper/2509.21043/) (2025)
- [CREATE: Testing LLMs for Associative Creativity](https://mlsanity.com/paper/2603.09970/) (2026)
- [Measuring LLM Novelty As The Frontier Of Original And High-Quality Output](https://mlsanity.com/paper/2504.09389/) (2025)

## Abstract (arXiv)

Large Language Models have consistently demonstrated a lack of creativity and diversity across tasks. Prior work has focused on addressing whether models are capable of generating creative outputs. Here, we aim to consider novelty and investigate what makes model-generated content novel or not novel in a task-specific manner. We propose a fine-grained evaluation metric GENIE to measure the novelty of responses along task-specific features with respect to a population of responses. We show that unlike GENIE, holistic metrics struggle to capture the high-dimensionality of novelty and do not provide insight on which properties they target. Finally, we use GENIE to measure the effectiveness of mitigation methods that address creativity to better understand where these methods can improve novelty.