# Eliciting the Priors of Large Language Models using Iterated In-Context Learning

arXiv: https://arxiv.org/abs/2406.01860 · HTML: https://mlsanity.com/paper/2406.01860/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 43 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [An Explanation of In-context Learning as Implicit Bayesian Inference](https://mlsanity.com/paper/2111.02080/) (2021)
- [Capabilities of GPT-4 on Medical Challenge Problems](https://mlsanity.com/paper/2303.13375/) (2023)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Data Interpreter: An LLM Agent For Data Science](https://mlsanity.com/paper/2402.18679/) (2024)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [Countering Language Drift with Seeded Iterated Learning](https://mlsanity.com/paper/2003.12694/) (2020)
- [Managing extreme AI risks amid rapid progress](https://mlsanity.com/paper/2310.17688/) (2023)
- [Thousands of AI Authors on the Future of AI](https://mlsanity.com/paper/2401.02843/) (2024)
- [Automated Statistical Model Discovery with Language Models](https://mlsanity.com/paper/2402.17879/) (2024)
- [Iterated learning for emergent systematicity in VQA](https://mlsanity.com/paper/2105.01119/) (2021)
- [Multi-label Iterated Learning for Image Classification with Label Ambiguity](https://mlsanity.com/paper/2111.12172/) (2021)
- [Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddings](https://mlsanity.com/paper/2310.18777/) (2023)
- [Imitation versus Innovation: What children can do that large language and language-and-vision models cannot (yet)?](https://mlsanity.com/paper/2305.07666/) (2023)
- [Supervised Seeded Iterated Learning for Interactive Language Learning](https://mlsanity.com/paper/2010.02975/) (2020)
- [Deep de Finetti: Recovering Topic Distributions from Large Language Models](https://mlsanity.com/paper/2312.14226/) (2023)
- [Recovering Mental Representations from Large Language Models with Markov Chain Monte Carlo](https://mlsanity.com/paper/2401.16657/) (2024)

## Built on by (most cited first, 19 of 19)

- [When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings](https://mlsanity.com/paper/2407.04503/) (2024)
- [Are Large Language Models Reliable AI Scientists? Assessing Reverse-Engineering of Black-Box Systems](https://mlsanity.com/paper/2505.17968/) (2025)
- [Levels of Analysis for Large Language Models](https://mlsanity.com/paper/2503.13401/) (2025)
- [When Agents Say One Thing and Do Another: Validating Elicited Beliefs from LLMs](https://mlsanity.com/paper/2602.06286/) (2026)
- [Chain of Thought Still Thinks Fast: APriCoT Helps with Thinking Slow](https://mlsanity.com/paper/2408.08651/) (2024)
- [Reflection-Bench: Evaluating Epistemic Agency in Large Language Models](https://mlsanity.com/paper/2410.16270/) (2024)
- [Steering Risk Preferences in Large Language Models by Aligning Behavioral and Neural Representations](https://mlsanity.com/paper/2505.11615/) (2025)
- [Language-Informed Synthesis of Rational Agent Models for Grounded Theory-of-Mind Reasoning On-The-Fly](https://mlsanity.com/paper/2506.16755/) (2025)
- [The Other Mind: How Language Models Exhibit Human Temporal Cognition](https://mlsanity.com/paper/2507.15851/) (2025)
- [Meta-learning ecological priors from large language models explains human learning and decision making](https://mlsanity.com/paper/2509.00116/) (2025)
- [Do Language Models Have Bayesian Brains? Distinguishing Stochastic and Deterministic Decision Patterns within Large Language Models](https://mlsanity.com/paper/2506.10268/) (2025)
- [Enhancing Zero-Shot Image Recognition in Vision-Language Models through Human-like Concept Guidance](https://mlsanity.com/paper/2503.15886/) (2025)
- [Leveraging Large Language Models in Visual Speech Recognition: Model Scaling, Context-Aware Decoding, and Iterative Polishing](https://mlsanity.com/paper/2506.02012/) (2025)
- [LLM-BI: Towards Fully Automated Bayesian Inference with Large Language Models](https://mlsanity.com/paper/2508.08300/) (2025)
- [Evolution and compression in LLMs: On the emergence of human-aligned categorization](https://mlsanity.com/paper/2509.08093/) (2025)
- [Probing Perceptual Priors of MLLMs via Gibbs Sampling with Interpretable Generative Controls](https://mlsanity.com/paper/2608.27727/) (2026)
- [Guided Data Generation for Understanding Model Behavior](https://mlsanity.com/paper/2502.06658/) (2025)
- [Eliciting Trustworthiness Priors of Large Language Models via Economic Games](https://mlsanity.com/paper/2602.00769/) (2026)
- [PriorProbe: Recovering Individual-Level Priors for Personalizing Neural Networks in Facial Expression Recognition](https://mlsanity.com/paper/2602.03882/) (2026)

## Abstract (arXiv)

As Large Language Models (LLMs) are increasingly deployed in real-world settings, understanding the knowledge they implicitly use when making decisions is critical. One way to capture this knowledge is in the form of Bayesian prior distributions. We develop a prompt-based workflow for eliciting prior distributions from LLMs. Our approach is based on iterated learning, a Markov chain Monte Carlo method in which successive inferences are chained in a way that supports sampling from the prior distribution. We validated our method in settings where iterated learning has previously been used to estimate the priors of human participants -- causal learning, proportion estimation, and predicting everyday quantities. We found that priors elicited from GPT-4 qualitatively align with human priors in these settings. We then used the same method to elicit priors from GPT-4 for a variety of speculative events, such as the timing of the development of superhuman AI.