mlsanityData to 7 Oct 2026

Chronologic: Measuring Language Models' Ability to Represent the Past

arXiv 2609.23178 ↗First posted 19 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its responses fit the period represented. Validation is challenging, because this is not a task living people ordinarily perform, and because many questions have multiple correct answers. We use historical texts to develop a benchmark for a model's representation of English-language contexts 1831-1930, relying on pairwise comparisons to multiple ground truths and strong distractors to score the hardest questions in an appropriately graduated way. We find that generative tasks are harder than discriminative ones; in fact, reasoning models can typically discern the weakness of their own generated answers. While models pretrained exclusively on historical text lead the pack when evaluated by answer likelihood, they cannot compete with commercial models in free generation. None of the models we tested represent historical contexts in a fully persuasive way yet, but progress toward that goal is evident.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 46 references that are in the map. First posted on arXiv 19 Sep 2026.

Builds on

15 of 46 references.

References · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 166
  5. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  6. 2024CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologiescited by 98
  7. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 97
  8. 2025Answer Matching Outperforms Multiple Choice for Language Model Evaluationcited by 37
  9. 2025Can Language Models Represent the Past without Anachronism?cited by 12
  10. 2025CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMscited by 11
  11. 2025Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usabilitycited by 7
  12. 2026Mediocrity is the key for LLM as a Judge Anchor Selectioncited by 6
3 more
  1. 2026Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoringcited by 2
  2. 2026JP-TL-Bench: Anchored Pairwise LLM Evaluation for Bidirectional Japanese-English Translationcited by 1
  3. 2026A Language Model from 1913: Pretraining on Historical Textcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023If the Sources Could Talk: Evaluating Large Language Models for Research Assistance in History
  2. 2024Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models
  3. 2024Time Awareness in Large Language Models: Benchmarking Fact Recall Across Time
  4. 2025"Lost-in-the-Later": Framework for Quantifying Contextual Grounding in Large Language Models
  5. 2025Kongzi: A Historical Large Language Model with Fact Enhancement
  6. 2025On Path to Multimodal Historical Reasoning: HistBench and HistAgent
  7. 2025Question Answering under Temporal Conflict: Evaluating and Organizing Evolving Knowledge with LLMs
  8. 2025A Study into Investigating Temporal Robustness of LLMs