# Time to Revist Exact Match

arXiv: https://arxiv.org/abs/2509.16720 · HTML: https://mlsanity.com/paper/2509.16720/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 13 of 13)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [Libra: Leveraging Temporal Images for Biomedical Radiology Analysis](https://mlsanity.com/paper/2411.19378/) (2024)
- [TempTabQA: Temporal Question Answering for Semi-Structured Tables](https://mlsanity.com/paper/2311.08002/) (2023)
- [Timo: Towards Better Temporal Reasoning for Language Models](https://mlsanity.com/paper/2406.14192/) (2024)
- [A better measure of relative prediction accuracy for model selection and model estimation](https://mlsanity.com/paper/2105.05249/) (2021)
- [ChronoSense: Exploring Temporal Understanding in Large Language Models with Time Intervals of Events](https://mlsanity.com/paper/2501.03040/) (2025)
- [A Study into Investigating Temporal Robustness of LLMs](https://mlsanity.com/paper/2503.17073/) (2025)
- [Temporal Knowledge Graph Question Answering: A Survey](https://mlsanity.com/paper/2406.14191/) (2024)
- [Enhancing Temporal Understanding in LLMs for Semi-structured Tables](https://mlsanity.com/paper/2407.16030/) (2024)
- [Towards Effective Time-Aware Language Representation: Exploring Enhanced Temporal Understanding in Language Models](https://mlsanity.com/paper/2406.01863/) (2024)
- [Factual Knowledge in Language Models: Robustness and Anomalies under Simple Temporal Context Variations](https://mlsanity.com/paper/2502.01220/) (2025)

## Abstract (arXiv)

Temporal question answering is an established method for evaluating temporal reasoning in large language models. Expected answers are often numeric (e.g., dates or durations), yet model responses are evaluated like regular text with exact match (EM), unable to distinguish small from large errors. In this investigative work, we frame temporal question answering as a numerical estimation task to assess the shortcomings of EM. We introduce TempAnswerQA, a benchmark distilled from Test of Time and TempTabQA, where all questions require a numerical, temporal answer, allowing us to evaluate models beyond EM. We use the forecasting metrics symmetric mean absolute percentage error (sMAPE) and mean absolute scaled error (MASE). With sMAPE, we find that error size and EM are decoupled. Models with low EM still have low sMAPE (both ~20%), and some models have high sMAPE despite high EM. Scaling errors by the deviation of the ground truth data with MASE reshuffles model rankings compared to EM, revealing gaps in models' understanding of temporal domain knowledge, especially when trained with synthetic data. Lastly, the models' most frequent error is to deviate by only ±1 from the ground truth. sMAPE and MASE, unlike EM, adequately weight these errors. Our findings underscore the need for specialised metrics for temporal QA tasks. Code and data are available on https://github.com/aauss/temporal-answer-qa.