# Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Based Calibrated Prior

arXiv: https://arxiv.org/abs/2506.03444 · HTML: https://mlsanity.com/paper/2506.03444/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 35 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Hypothesis Generation with Large Language Models](https://mlsanity.com/paper/2404.04326/) (2024)
- [LMPriors: Pre-Trained Language Models as Task-Specific Priors](https://mlsanity.com/paper/2210.12530/) (2022)
- [LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language](https://mlsanity.com/paper/2405.12856/) (2024)
- [Improving Scientific Hypothesis Generation with Knowledge Grounded Large Language Models](https://mlsanity.com/paper/2411.02382/) (2024)
- [AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling](https://mlsanity.com/paper/2411.17284/) (2024)
- [Correlation Sketches for Approximate Join-Correlation Queries](https://mlsanity.com/paper/2104.03353/) (2021)
- [Data Polygamy: The Many-Many Relationships among Urban Spatio-Temporal Data Sets](https://mlsanity.com/paper/1610.06978/) (2016)
- [Efficiently Estimating Mutual Information Between Attributes Across Tables](https://mlsanity.com/paper/2403.15553/) (2024)

## Built on by (most cited first, 1 of 1)

- [Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics](https://mlsanity.com/paper/2510.09901/) (2025)

## Abstract (arXiv)

As hypothesis generation becomes increasingly automated, a new bottleneck has emerged: hypothesis assessment. Modern systems can surface thousands of statistical relationships-correlations, trends, causal links-but offer little guidance on which ones are novel, non-trivial, or worthy of expert attention. In this work, we study the complementary problem to hypothesis generation: automatic hypothesis assessment. Specifically, we ask: given a large set of statistical relationships, can we automatically assess which ones are novel and worth further exploration? We focus on correlations as they are a common entry point in exploratory data analysis that often serve as the basis for forming deeper scientific or causal hypotheses. To support automatic assessment, we propose to leverage the vast knowledge encoded in LLMs' weights to derive a prior distribution over the correlation value of a variable pair. If an LLM's prior expects the correlation value observed, then such correlation is not surprising, and vice versa. We propose the Logit-based Calibrated Prior, an LLM-elicited correlation prior that transforms the model's raw output logits into a calibrated, continuous predictive distribution over correlation values. We evaluate the prior on a benchmark of 2,096 real-world variable pairs and it achieves a sign accuracy of 78.8%, a mean absolute error of 0.26, and 95% credible interval coverage of 89.2% in predicting Pearson correlation coefficient. It also outperforms a fine-tuned RoBERTa classifier in binary correlation prediction and achieves higher precision@K in hypothesis ranking. We further show that the prior generalizes to correlations not seen during LLM pretraining, reflecting context-sensitive reasoning rather than memorization.