mlsanityData to 7 Oct 2026

When Lexical Change Misleads: Rethinking Dynamic Topic Model Evaluation with Traditional and LLM-Based Metrics

arXiv 2608.13835 ↗First posted 13 Aug 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Dynamic topic models capture evolving word distributions, but traditional coherence metrics may fail when vocabulary changes while semantic meaning persists. We evaluate 120 topics from CoNTM and DLDA across NYT, DBLP, and arXiv, using three human annotators and Low, Medium, and High lexical-change categories. Traditional temporal coherence shows highly variable agreement with human judgments (ρ=-0.256 to 0.614). In contrast, LLM-based semantic similarity agrees strongly with human semantic judgments for CoNTM on NYT (ρ=0.609), DBLP (ρ=0.721), and arXiv (ρ=0.502), but is less consistent for DLDA. Lexical-change stratification reveals variation hidden by aggregate evaluation. We therefore advocate lexical-change-aware evaluation, jointly reporting traditional coherence and LLM-based semantic measures as complementary rather than interchangeable signals.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 1 of its 13 references that are in the map. First posted on arXiv 13 Aug 2026.

Builds on

1 of 13 references.

References · 1

  1. 2021Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherencecited by 74

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Metrics for Lexical Semantic Change Detection
  2. 2024Semantic Change Characterization with LLMs using Rhetorics
  3. 2026Measuring Collective Semantic Change in Populations of Language Model Agents
  4. 2024Can large language models understand uncommon meanings of common words?
  5. 2024Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time
  6. 2026Fully Automated Identification of Lexical Alignment and Preference-Stage Shifts in Large Language Models
  7. 2026RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World
  8. 2023Rethinking STS and NLI in Large Language Models