mlsanityData to 7 Oct 2026

MuseScorer: Idea Originality Scoring At Scale

arXiv 2505.16232 ↗First posted 22 May 2025cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

An objective, face-valid method for scoring idea originality is to measure each idea's statistical infrequency within a population -- an approach long used in creativity research. Yet, computing these frequencies requires manually bucketing idea rephrasings, a process that is subjective, labor-intensive, error-prone, and brittle at scale. We introduce MuseScorer, a fully automated, psychometrically validated system for frequency-based originality scoring. MuseScorer integrates a Large Language Model (LLM) with externally orchestrated retrieval: given a new idea, it retrieves semantically similar prior idea-buckets and zero-shot prompts the LLM to judge whether the idea fits an existing bucket or forms a new one. These buckets enable frequency-based originality scoring without human annotation. Across five datasets N_{participants}=1143, n_{ideas}=16,294), MuseScorer matches human annotators in idea clustering structure (AMI = 0.59) and participant-level scoring (r = 0.89), while demonstrating strong convergent and external validity. The system enables scalable, intent-sensitive, and human-aligned originality assessment for creativity research.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 19 of its 77 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

19 of 77 references.

References · 19

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  5. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 995
  6. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  7. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  8. 2020Distilling Knowledge from Reader to Retriever for Question Answeringcited by 226
  9. 2023Art or Artifice? Large Language Models and the False Promise of Creativitycited by 168
  10. 2023Generative artificial intelligence enhances creativity but reduces the diversity of novel contentcited by 165
  11. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  12. 2023Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Codingcited by 82
7 more
  1. 2023Large Language Models Enable Few-Shot Clusteringcited by 72
  2. 2023LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysiscited by 70
  3. 2023LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Codingcited by 64
  4. 2024DALK: Dynamic Co-Augmentation of LLMs and KG to answer Alzheimer's Disease Questions with Scientific Literaturecited by 50
  5. 2024Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-AI Alignment in the Writing Process through Editscited by 49
  6. 2024Comparative Analysis of Listwise Reranking with Large Language Models in Limited-Resource Language Contextscited by 7
  7. 2024AI Can Enhance Creativity in Social Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Ex Ante Evaluation of AI-Induced Idea Diversity Collapsecited by 1
  2. 2026Anchorless Diversification for Parallel LLM Ideationcited by 0