mlsanityData to 7 Oct 2026

Surprisal and Metaphor Novelty Judgments: Moderate Correlations and Divergent Scaling Effects Revealed by Corpus-Based and Synthetic Datasets

arXiv 2601.02015 ↗First posted 5 Jan 2026cs.CL cs.AI cs.ITcited by 1

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Novel metaphor comprehension involves complex semantic processes and linguistic creativity, making it an interesting task for studying language models (LMs). This study investigates whether surprisal, a probabilistic measure of predictability in LMs, correlates with annotations of metaphor novelty in different datasets. We analyse the surprisal of metaphoric words in corpus-based and synthetic metaphor datasets using 16 causal LM variants. We propose a cloze-style surprisal method that conditions on full-sentence context. Results show that LM surprisal yields significant moderate correlations with scores/labels of metaphor novelty. We further identify divergent scaling patterns: on corpus-based data, correlation strength decreases with model size (inverse scaling effect), whereas on synthetic data it increases (quality-power hypothesis). We conclude that while surprisal can partially account for annotations of metaphor novelty, it remains limited as a metric of linguistic creativity. Code and data are publicly available: https://github.com/OmarMomen14/surprisal-metaphor-novelty

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 16 of its 60 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

16 of 60 references.

References · 16

  1. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  2. 2020On the Predictive Power of Neural Language Models for Human Real-Time Comprehension Behaviorcited by 115
  3. 2022Why Does Surprisal From Larger Transformer-Based Language Models Provide a Poorer Fit to Human Reading Times?cited by 111
  4. 2017Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotationcited by 77
  5. 2021Revisiting the Uniform Information Density Hypothesiscited by 63
  6. 2023Transformer-Based Language Model Surprisal Predicts Human Reading Times Best with About Two Billion Training Tokenscited by 39
  7. 2024Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Timescited by 37
  8. 2024How to Compute the Probability of a Wordcited by 36
  9. 2022Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languagescited by 30
  10. 2024Metaphor Understanding Challenge Dataset for LLMscited by 30
  11. 2024Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilitiescited by 30
  12. 2025Causal Estimation of Tokenisation Biascited by 16
4 more
  1. 2023Large Language Model Displays Emergent Ability to Interpret Novel Literary Metaphorscited by 12
  2. 2024What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Lengthcited by 10
  3. 2022Metaphorical Polysemy Detection: Conventional Metaphor meets Word Sense Disambiguationcited by 6
  4. 2022Leveraging a New Spanish Corpus for Multilingual and Crosslingual Metaphor Detectioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Frequency Confound in Language-Model Surprisal and Metaphor Noveltycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Probing Semantic Alignment, Lexical Invariance, and Syntactic Influence in LLM Metaphor Processing
  2. 2024Science is Exploration: Computational Frontiers for Conceptual Metaphor Theory
  3. 2022Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models
  4. 2025Metaphor identification using large language models: A comparison of RAG, prompt engineering, and fine-tuning
  5. 2026Metaphors are a Source of Cross-Domain Misalignment of Large Reasoning Models
  6. 2026Metaphors We Compute By: A Computational Audit of Cultural Translation vs. Thinking in LLMs
  7. 2024A Dual-Perspective Metaphor Detection Framework Using Large Language Models
  8. 2023I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors