# Structured Inference with Large Language Gibbs

arXiv: https://arxiv.org/abs/2606.19264 · HTML: https://mlsanity.com/paper/2606.19264/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 50 references that are in the map.
- First posted on arXiv 17 Jun 2026.

## Builds on (most cited first, 28 of 28)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity](https://mlsanity.com/paper/2104.08786/) (2021)
- [Language Models are Realistic Tabular Data Generators](https://mlsanity.com/paper/2210.06280/) (2022)
- [Learning Bayesian Networks: The Combination of Knowledge and Statistical Data](https://mlsanity.com/paper/1302.6815/) (2013)
- [BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model](https://mlsanity.com/paper/1902.04094/) (2019)
- [COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics](https://mlsanity.com/paper/2202.11705/) (2022)
- [Bayesian Structure Learning with Generative Flow Networks](https://mlsanity.com/paper/2202.13903/) (2022)
- [CGMH: Constrained Sentence Generation by Metropolis-Hastings Sampling](https://mlsanity.com/paper/1811.10996/) (2018)
- [Mix and Match: Learning-free Controllable Text Generation using Energy Language Models](https://mlsanity.com/paper/2203.13299/) (2022)
- [Gradient-Based Constrained Sampling from Language Models](https://mlsanity.com/paper/2205.12558/) (2022)
- [All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning](https://mlsanity.com/paper/2503.01067/) (2025)
- [Reprompting: Automated Chain-of-Thought Prompt Inference Through Gibbs Sampling](https://mlsanity.com/paper/2305.09993/) (2023)
- [Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings](https://mlsanity.com/paper/2106.02736/) (2021)
- [Towards Scalable Bayesian Learning of Causal DAGs](https://mlsanity.com/paper/2010.00684/) (2020)
- [Large Language Models are Effective Priors for Causal Graph Discovery](https://mlsanity.com/paper/2405.13551/) (2024)
- [QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine Translation](https://mlsanity.com/paper/2406.00049/) (2024)
- [AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling](https://mlsanity.com/paper/2411.17284/) (2024)
- [Large Language Bayes](https://mlsanity.com/paper/2504.14025/) (2025)
- [Deriving Language Models from Masked Language Models](https://mlsanity.com/paper/2305.15501/) (2023)
- [bnRep: A repository of Bayesian networks from the academic literature](https://mlsanity.com/paper/2409.19158/) (2024)
- [The Illusion of Stochasticity in LLMs](https://mlsanity.com/paper/2604.06543/) (2026)
- [Flipping Against All Odds: Reducing LLM Coin Flip Bias via Verbalized Rejection Sampling](https://mlsanity.com/paper/2506.09998/) (2025)
- [Large (Vision) Language Models are Unsupervised In-Context Learners](https://mlsanity.com/paper/2504.02349/) (2025)
- [Principled Gradient-based Markov Chain Monte Carlo for Text Generation](https://mlsanity.com/paper/2312.17710/) (2023)
- [How many patients could we save with LLM priors?](https://mlsanity.com/paper/2509.04250/) (2025)

## Abstract (arXiv)

The knowledge encoded in large language models (LLMs) can serve as a substrate for structured reasoning over variables describing a complex world, but accessing this knowledge in a probabilistically coherent manner poses a difficult inference problem. We propose Large Language Gibbs, a scheme for structured probabilistic inference that uses conditional distributions of an LLM as transition operators. Rather than sampling structured objects through single-pass autoregressive generation, we iteratively resample individual variables conditioned on others using an LLM's next-token conditionals. This approach avoids order-dependent biases and produces a stationary distribution that reflects a compromise between all local conditionals. We apply this approach to sampling from synthetic distributions, consistent reasoning tasks, and Bayesian structure learning. The results suggest that the use of LLM conditionals in MCMC is a practical alternative to one-pass generation for structured probabilistic inference under a world prior accessible through noisy LLM conditionals.