# Large Language Models for Predictive Analysis: How Far Are They?

arXiv: https://arxiv.org/abs/2505.17149 · HTML: https://mlsanity.com/paper/2505.17149/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 50 references that are in the map.
- First posted on arXiv 22 May 2025.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Phi-4 Technical Report](https://mlsanity.com/paper/2412.08905/) (2024)
- [LLM Evaluators Recognize and Favor Their Own Generations](https://mlsanity.com/paper/2404.13076/) (2024)
- [Understanding the Effects of RLHF on LLM Generalisation and Diversity](https://mlsanity.com/paper/2310.06452/) (2023)
- [Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation](https://mlsanity.com/paper/2308.15363/) (2023)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [Large Language Models are Inconsistent and Biased Evaluators](https://mlsanity.com/paper/2405.01724/) (2024)
- [Large Language Models are few(1)-shot Table Reasoners](https://mlsanity.com/paper/2210.06710/) (2022)
- [An Evaluation of Change Point Detection Algorithms](https://mlsanity.com/paper/2003.06222/) (2020)
- [ChatGLM-Math: Improving Math Problem-Solving in Large Language Models with a Self-Critique Pipeline](https://mlsanity.com/paper/2404.02893/) (2024)
- [Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries](https://mlsanity.com/paper/2312.13671/) (2023)
- [Large Language Models as Data Preprocessors](https://mlsanity.com/paper/2308.16361/) (2023)
- [Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions](https://mlsanity.com/paper/2408.08781/) (2024)
- [Large Language Model Prediction Capabilities: Evidence from a Real-World Forecasting Tournament](https://mlsanity.com/paper/2310.13014/) (2023)
- [Querying Large Language Models with SQL](https://mlsanity.com/paper/2304.00472/) (2023)
- [Can Language Models Use Forecasting Strategies?](https://mlsanity.com/paper/2406.04446/) (2024)
- [Humans vs Large Language Models: Judgmental Forecasting in an Era of Advanced AI](https://mlsanity.com/paper/2312.06941/) (2023)
- [DAGPrompT: Pushing the Limits of Graph Prompting with a Distribution-aware Graph Prompt Tuning Approach](https://mlsanity.com/paper/2501.15142/) (2025)
- [HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position](https://mlsanity.com/paper/2403.19115/) (2024)
- [TreeMAN: Tree-enhanced Multimodal Attention Network for ICD Coding](https://mlsanity.com/paper/2305.18576/) (2023)

## Built on by (most cited first, 3 of 3)

- [Spectral Insights into Data-Oblivious Critical Layers in Large Language Models](https://mlsanity.com/paper/2506.00382/) (2025)
- [Representation Interventions Enable Lifelong Knowledge Memory Control in LLMs](https://mlsanity.com/paper/2511.20892/) (2025)
- [SheetDesigner: MLLM-Powered Spreadsheet Layout Generation with Rule-Based and Vision-Based Reflection](https://mlsanity.com/paper/2509.07473/) (2025)

## Abstract (arXiv)

Predictive analysis is a cornerstone of modern decision-making, with applications in various domains. Large Language Models (LLMs) have emerged as powerful tools in enabling nuanced, knowledge-intensive conversations, thus aiding in complex decision-making tasks. With the burgeoning expectation to harness LLMs for predictive analysis, there is an urgent need to systematically assess their capability in this domain. However, there is a lack of relevant evaluations in existing studies. To bridge this gap, we introduce the PredictiQ benchmark, which integrates 1130 sophisticated predictive analysis queries originating from 44 real-world datasets of 8 diverse fields. We design an evaluation protocol considering text analysis, code generation, and their alignment. Twelve renowned LLMs are evaluated, offering insights into their practical use in predictive analysis. Generally, we believe that existing LLMs still face considerable challenges in conducting predictive analysis. See hrefhttps://github.com/Cqkkkkkk/PredictiQGithub.