# Chronologic: Measuring Language Models' Ability to Represent the Past

arXiv: https://arxiv.org/abs/2609.23178 · HTML: https://mlsanity.com/paper/2609.23178/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 46 references that are in the map.
- First posted on arXiv 19 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Towards Measuring and Modeling "Culture" in LLMs: A Survey](https://mlsanity.com/paper/2403.15412/) (2024)
- [Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4](https://mlsanity.com/paper/2305.00118/) (2023)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [CultureBank: An Online Community-Driven Knowledge Base Towards Culturally Aware Language Technologies](https://mlsanity.com/paper/2404.15238/) (2024)
- [Answer Matching Outperforms Multiple Choice for Language Model Evaluation](https://mlsanity.com/paper/2507.02856/) (2025)
- [Can Language Models Represent the Past without Anachronism?](https://mlsanity.com/paper/2505.00030/) (2025)
- [CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs](https://mlsanity.com/paper/2509.16188/) (2025)
- [Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability](https://mlsanity.com/paper/2506.08300/) (2025)
- [Mediocrity is the key for LLM as a Judge Anchor Selection](https://mlsanity.com/paper/2603.16848/) (2026)
- [Question Difficulty Estimation for Large Language Models via Answer Plausibility Scoring](https://mlsanity.com/paper/2605.12398/) (2026)
- [A Language Model from 1913: Pretraining on Historical Text](https://mlsanity.com/paper/2606.02991/) (2026)
- [JP-TL-Bench: Anchored Pairwise LLM Evaluation for Bidirectional Japanese-English Translation](https://mlsanity.com/paper/2601.00223/) (2026)

## Abstract (arXiv)

Language models are appealing tools for research on the past. But to trust the evidence a model provides, researchers need to know whether its responses fit the period represented. Validation is challenging, because this is not a task living people ordinarily perform, and because many questions have multiple correct answers. We use historical texts to develop a benchmark for a model's representation of English-language contexts 1831-1930, relying on pairwise comparisons to multiple ground truths and strong distractors to score the hardest questions in an appropriately graduated way. We find that generative tasks are harder than discriminative ones; in fact, reasoning models can typically discern the weakness of their own generated answers. While models pretrained exclusively on historical text lead the pack when evaluated by answer likelihood, they cannot compete with commercial models in free generation. None of the models we tested represent historical contexts in a fully persuasive way yet, but progress toward that goal is evident.