# AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling

arXiv: https://arxiv.org/abs/2411.17284 · HTML: https://mlsanity.com/paper/2411.17284/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 70 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Transformers learn in-context by gradient descent](https://mlsanity.com/paper/2212.07677/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [TabLLM: Few-shot Classification of Tabular Data with Large Language Models](https://mlsanity.com/paper/2210.10723/) (2022)
- [Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning](https://mlsanity.com/paper/2301.11916/) (2023)
- [Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering](https://mlsanity.com/paper/2305.03403/) (2023)
- [Empowering Biomedical Discovery with AI Agents](https://mlsanity.com/paper/2404.02831/) (2024)
- [AutoML-GPT: Automatic Machine Learning with GPT](https://mlsanity.com/paper/2305.02499/) (2023)
- [A Review of Challenges and Opportunities in Machine Learning for Health](https://mlsanity.com/paper/1806.00388/) (2018)
- [MLCopilot: Unleashing the Power of Large Language Models in Solving Machine Learning Tasks](https://mlsanity.com/paper/2304.14979/) (2023)
- [Scaling Scaling Laws with Board Games](https://mlsanity.com/paper/2104.03113/) (2021)
- [Is In-Context Learning in Large Language Models Bayesian? A Martingale Perspective](https://mlsanity.com/paper/2406.00793/) (2024)
- [LMPriors: Pre-Trained Language Models as Task-Specific Priors](https://mlsanity.com/paper/2210.12530/) (2022)
- [LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language](https://mlsanity.com/paper/2405.12856/) (2024)
- [Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers](https://mlsanity.com/paper/1805.08154/) (2018)
- [Automated Statistical Model Discovery with Language Models](https://mlsanity.com/paper/2402.17879/) (2024)
- [Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models](https://mlsanity.com/paper/2404.06209/) (2024)
- [Had enough of experts? Quantitative knowledge retrieval from large language models](https://mlsanity.com/paper/2402.07770/) (2024)

## Built on by (most cited first, 15 of 15)

- [Extracting Probabilistic Knowledge from Large Language Models for Bayesian Network Parameterization](https://mlsanity.com/paper/2505.15918/) (2025)
- [Steering Risk Preferences in Large Language Models by Aligning Behavioral and Neural Representations](https://mlsanity.com/paper/2505.11615/) (2025)
- [How many patients could we save with LLM priors?](https://mlsanity.com/paper/2509.04250/) (2025)
- [Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Based Calibrated Prior](https://mlsanity.com/paper/2506.03444/) (2025)
- [LLM-BI: Towards Fully Automated Bayesian Inference with Large Language Models](https://mlsanity.com/paper/2508.08300/) (2025)
- [ANCHOR: Abductive Network Construction with Hierarchical Orchestration for Reliable Probability Inference in Large Language Models](https://mlsanity.com/paper/2605.10328/) (2026)
- [Using Large Language Models to Suggest Informative Prior Distributions in Bayesian Statistics](https://mlsanity.com/paper/2506.21964/) (2025)
- [Score-Based Density Estimation from Pairwise Comparisons](https://mlsanity.com/paper/2510.09146/) (2025)
- [What Language Models Know But Don't Say: Non-Generative Prior Extraction for Generalization](https://mlsanity.com/paper/2601.17609/) (2026)
- [Language-Induced Priors for Domain Adaptation](https://mlsanity.com/paper/2605.14301/) (2026)
- [AI4BayesCode: From Natural Language Descriptions to Validated Modular Stateful Bayesian Samplers](https://mlsanity.com/paper/2605.18476/) (2026)
- [Supervised Learning Objectives for High-Dimensional Treatment Effects](https://mlsanity.com/paper/2605.27281/) (2026)
- [Structured Inference with Large Language Gibbs](https://mlsanity.com/paper/2606.19264/) (2026)
- [Generative Priors Conditioned on Natural Language for Bayesian Inversion in PDEs](https://mlsanity.com/paper/2609.32941/) (2026)
- [Marginal Response Surface Elicitation for Zero-Label Tabular Learning](https://mlsanity.com/paper/2609.39639/) (2026)

## Abstract (arXiv)

Large language models (LLMs) acquire a breadth of information across various domains. However, their computational complexity, cost, and lack of transparency often hinder their direct application for predictive tasks where privacy and interpretability are paramount. In fields such as healthcare, biology, and finance, specialised and interpretable linear models still hold considerable value. In such domains, labelled data may be scarce or expensive to obtain. Well-specified prior distributions over model parameters can reduce the sample complexity of learning through Bayesian inference; however, eliciting expert priors can be time-consuming. We therefore introduce AutoElicit to extract knowledge from LLMs and construct priors for predictive models. We show these priors are informative and can be refined using natural language. We perform a careful study contrasting AutoElicit with in-context learning and demonstrate how to perform model selection between the two methods. We find that AutoElicit yields priors that can substantially reduce error over uninformative priors, using fewer labels, and consistently outperform in-context learning. We show that AutoElicit saves over 6 months of labelling effort when building a new predictive model for urinary tract infections from sensor recordings of people living with dementia.