Predicting Language Models' Success at Zero-Shot Probabilistic Prediction
Not on the map. Why?
Abstract (arXiv)
Recent work has investigated the capabilities of large language models (LLMs) as zero-shot models for generating individual-level characteristics (e.g., to serve as risk models or augment survey datasets). However, when should a user have confidence that an LLM will provide high-quality predictions for their particular task? To address this question, we conduct a large-scale empirical study of LLMs' zero-shot predictive capabilities across a wide range of tabular prediction tasks. We find that LLMs' performance is highly variable, both on tasks within the same dataset and across different datasets. However, when the LLM performs well on the base prediction task, its predicted probabilities become a stronger signal for individual-level accuracy. Then, we construct metrics to predict LLMs' performance at the task level, aiming to distinguish between tasks where LLMs may perform well and where they are likely unsuitable. We find that some of these metrics, each of which are assessed without labeled data, yield strong signals of LLMs' predictive performance on new tasks.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 39 references that are in the map. First posted on arXiv 18 Sep 2025.
Builds on
26 of 39 references.
References · 26
- 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
- 2021Retiring Adult: New Datasets for Fair Machine Learningcited by 449
- 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
- 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
- 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 398
- 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 249
- 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 180
- 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
- 2024Language Models with Conformal Factuality Guaranteescited by 131
- 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
14 more
- 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 85
- 2024Large Scale Transfer Learning for Tabular Data via Language Modelingcited by 76
- 2020Understanding tables with intermediate pre-trainingcited by 68
- 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 66
- 2024Calibrating Large Language Models Using Their Generations Onlycited by 65
- 2024Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinationscited by 54
- 2024Multicalibration for Confidence Scoring in LLMscited by 46
- 2023Tab-CoT: Zero-shot Tabular Chain of Thoughtcited by 42
- 2023Large Language Models Are Zero-Shot Text Classifierscited by 35
- 2024Evaluating language models as risk scorescited by 27
- 2023Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classificationscited by 15
- 2024How Much Can We Forget about Data Contamination?cited by 14
- 2024Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretrainingcited by 12
- 2023UniPredict: Large Language Models are Universal Tabular Classifierscited by 9
Built on by
Citing papers per year: 2026: 2.
2 papers.