AI Predicts AGI: Leveraging AGI Forecasting and Peer Review to Explore LLMs' Complex Reasoning Capabilities
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
We tasked 16 state-of-the-art large language models (LLMs) with estimating the likelihood of Artificial General Intelligence (AGI) emerging by 2030. To assess the quality of these forecasts, we implemented an automated peer review process (LLM-PR). The LLMs' estimates varied widely, ranging from 3% (Reka- Core) to 47.6% (GPT-4o), with a median of 12.5%. These estimates closely align with a recent expert survey that projected a 10% likelihood of AGI by 2027, underscoring the relevance of LLMs in forecasting complex, speculative scenarios. The LLM-PR process demonstrated strong reliability, evidenced by a high Intraclass Correlation Coefficient (ICC = 0.79), reflecting notable consistency in scoring across the models. Among the models, Pplx-70b-online emerged as the top performer, while Gemini-1.5-pro-api ranked the lowest. A cross-comparison with external benchmarks, such as LMSYS Chatbot Arena, revealed that LLM rankings remained consistent across different evaluation methods, suggesting that existing benchmarks may not encapsulate some of the skills relevant for AGI prediction. We further explored the use of weighting schemes based on external benchmarks, optimizing the alignment of LLMs' predictions with human expert forecasts. This analysis led to the development of a new, 'AGI benchmark' designed to highlight performance differences in AGI-related tasks. Our findings offer insights into LLMs' capabilities in speculative, interdisciplinary forecasting tasks and emphasize the growing need for innovative evaluation frameworks for assessing AI performance in complex, uncertain real-world scenarios.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 17 references that are in the map. First posted on arXiv 12 Dec 2024.
Builds on
12 of 17 references.
References · 12
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
- 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
- 2024Inadequacies of Large Language Model Benchmarks in the Era of Generative Artificial Intelligencecited by 65
- 2023Metacognitive Prompting Improves Understanding in Large Language Modelscited by 58
- 2024Time Series Forecasting with LLMs: Understanding and Enhancing Model Capabilitiescited by 48
- 2024PRE: A Peer Review Based Large Language Model Evaluatorcited by 28
- 2023Beyond Static Datasets: A Deep Interaction Approach to LLM Evaluationcited by 24
- 2024AI-Augmented Predictions: LLM Assistants Improve Human Forecasting Accuracycited by 21
- 2024PiCO: Peer Review in LLMs based on the Consistency Optimizationcited by 14
- 2022Forecasting AI Progress: Evidence from a Survey of Machine Learning Researcherscited by 7
- 2023Post Turing: Mapping the landscape of LLM Evaluationcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
- 2026LLM-based Agents for Forecasting and Prediction: Methods, Training, Evaluation, and Applications
- 2025Towards New Benchmark for AI Alignment & Sentiment Analysis in Socially Important Issues: A Comparative Study of Human and LLMs in the Context of AGI
- 2025MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation
- 2024Heuristics and Biases in AI Decision-Making: Implications for Responsible AGI
- 2025Challenges and Applications of Large Language Models: A Comparison of GPT and DeepSeek family of models
- 2026LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
- 2025Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspective