mlsanityData to 7 Oct 2026

Consistency Checks for Language Model Forecasters

arXiv 2412.18544 ↗First posted 24 Dec 2024cs.LG cs.AI cs.CLcited by 27

Not on the map. Why?

Abstract (arXiv)

Forecasting is a task that is difficult to evaluate: the ground truth can only be known in the future. Recent work showing LLM forecasters rapidly approaching human-level performance begs the question: how can we benchmark and evaluate these forecasters instantaneously? Following the consistency check framework, we measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions. We propose a new, general consistency metric based on arbitrage: for example, if a forecasting AI illogically predicts that both the Democratic and Republican parties have 60% probability of winning the 2024 US presidential election, an arbitrageur can trade against the forecaster's predictions and make a profit. We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits the predictions of the forecaster, and measures the consistency of the predictions. We then build a standard, proper-scoring-rule forecasting benchmark, and show that our (instantaneous) consistency metrics correlate with LLM forecasters' ground truth Brier scores (which are only known in the future). We also release a consistency benchmark that resolves in 2028, providing a long-term evaluation tool for forecasting.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 24 Dec 2024.

Builds on

18 of 37 references.

References · 18

  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  4. 2018AI safety via debatecited by 384
  5. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  6. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  7. 2020Metamorphic Testing: A New Approach for Generating Next Test Casescited by 71
  8. 2024Wisdom of the Silicon Crowd: LLM Ensemble Prediction Capabilities Rival Human Crowd Accuracycited by 66
  9. 2022Forecasting Future World Events with Neural Networkscited by 63
  10. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  11. 2023Benchmarking and Improving Generator-Validator Consistency of Language Modelscited by 43
  12. 2023Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournamentcited by 24
6 more
  1. 2016Logical Inductioncited by 15
  2. 2020Decentralized Reinforcement Learning: Global Decision-Making via Local Economic Transactionscited by 14
  3. 2022Are AlphaZero-like Agents Robust to Adversarial Perturbations?cited by 14
  4. 2020Testing Monotonicity of Machine Learning Modelscited by 8
  5. 2022Specifying and Testing k-Safety Properties for Machine-Learning Modelscited by 5
  6. 2024Language Models Can Reduce Asymmetry in Information Marketscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 19.

27 papers.

Sort:

By area:

  1. 2025Empowering LLMs with Logical Reasoning: A Comprehensive Surveycited by 51
  2. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 36
  3. 2025Pitfalls in Evaluating Language Model Forecasterscited by 23
  4. 2025Scaling Open-Ended Reasoning to Predict the Futurecited by 19
  5. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  6. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
  7. 2025Are LLM Belief Updates Consistent with Bayes' Theorem?cited by 8
  8. 2025Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impactscited by 6
  9. 2025Adaptively profiling models with task elicitationcited by 4
  10. 2026Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theoremscited by 3
  11. 2026Dutch Books for Language Modelscited by 3
  12. 2026Reinforcement Learning for LLM-based Event Forecastingcited by 2
15 more
  1. 2026ForecastBench-Sim: A Simulated-World Forecasting Benchmarkcited by 2
  2. 2026Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Modelscited by 2
  3. 2026TruthTensor: Evaluating LLMs through Human Imitation on Prediction Market under Drift and Holistic Reasoningcited by 1
  4. 2026TimeSeek: Temporal Reliability of Agentic Forecasterscited by 1
  5. 2026Locally Coherent, Globally Incoherent: Bounding Compositional Incoherence in Multi-Component LLM Agentscited by 1
  6. 2026Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?cited by 1
  7. 2026QuantSightBench: Evaluating LLM Quantitative Forecasting with Prediction Intervalscited by 0
  8. 2026Global Merger-Arbitrage Forecasting with Language Modelscited by 0
  9. 2026OptimismBench: Forecasting Bias and the Alignment Effect in Language Model Judgmentcited by 0
  10. 2026LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applicationscited by 0
  11. 2026Forecast-Dojo: Replayable Environments for Benchmarking and Training LLM Forecasting Agentscited by 0
  12. 2026Can LLMs Predict the Future? A Brier Score Analysis of Prediction Marketscited by 0
  13. 2026Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecastscited by 0
  14. 2026CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inferencecited by 0
  15. 2026Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfacescited by 0