mlsanityData to 7 Oct 2026

Amortizing intractable inference in large language models

arXiv 2310.04363 ↗First posted 6 Oct 2023cs.LG cs.CLcited by 94

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Autoregressive large language models (LLMs) compress knowledge from their training data through next-token conditional distributions. This limits tractable querying of this knowledge to start-to-end autoregressive sampling. However, many tasks of interest -- including sequence continuation, infilling, and other forms of constrained generation -- involve sampling from intractable posterior distributions. We address this limitation by using amortized Bayesian inference to sample from these intractable posteriors. Such amortization is algorithmically achieved by fine-tuning LLMs via diversity-seeking reinforcement learning algorithms: generative flow networks (GFlowNets). We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training and reward-maximizing policy optimization. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem and demonstrate that our approach enables data-efficient adaptation of LLMs to tasks that require multi-step rationalization and tool use.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Builds on 76 of its 99 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

76 of 99 references.

References · 76

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  11. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
38 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  7. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 895
  8. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  10. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  11. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  12. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 618
  13. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  14. 2023Faith and Fate: Limits of Transformers on Compositionalitycited by 496
  15. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  16. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  17. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  18. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  19. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  20. 2021GFlowNet Foundationscited by 286
  21. 2022Teaching Small Language Models to Reasoncited by 279
  22. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  23. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  24. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
  25. 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
  26. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  27. 2022Biological Sequence Design with GFlowNetscited by 172
  28. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  29. 2021NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristicscited by 153
  30. 2022Bayesian Structure Learning with Generative Flow Networkscited by 153
  31. 2022Learning GFlowNets from partial episodes for improved convergence and stabilitycited by 129
  32. 2020Enabling Language Models to Fill in the Blankscited by 121
  33. 2023Teaching Arithmetic to Small Transformerscited by 121
  34. 2022Teaching Algorithmic Reasoning via In-context Learningcited by 111
  35. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  36. 2022Language Model Cascadescited by 104
  37. 2022GFlowNets and variational inferencecited by 103
  38. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 35; 2025: 27; 2026: 32.

94 papers.

Sort:

By area:

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  2. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  3. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 118
  4. 2025Self-Evolving Curriculum for LLM Reasoningcited by 107
  5. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
  6. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  7. 2026Privileged Information Distillation for Language Modelscited by 79
  8. 2024Amortizing intractable inference in diffusion models for vision, language, and controlcited by 70
  9. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  10. 2024Learning diverse attacks on large language models for robust red-teaming and safety tuningcited by 54
  11. 2024Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewardingcited by 53
  12. 2024Discrete Probabilistic Inference as Control in Multi-path Environmentscited by 40
38 more
  1. 2025FlowRL: Matching Reward Distributions for LLM Reasoningcited by 38
  2. 2024Adaptable Logical Control for Large Language Modelscited by 30
  3. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  4. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  5. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 29
  6. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 28
  7. 2025Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Trainingcited by 27
  8. 2025Latent Thought Models with Variational Bayes Inference-Time Computationcited by 25
  9. 2024Adaptive teachers for amortized samplerscited by 23
  10. 2025From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generationcited by 20
  11. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 20
  12. 2024BIRD: A Trustworthy Bayesian Inference Framework for Large Language Modelscited by 18
  13. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 16
  14. 2025GFlowVLM: Enhancing Multi-step Reasoning in Vision-Language Models with Generative Flow Networkscited by 15
  15. 2024ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domaincited by 14
  16. 2024Pessimistic Backward Policy for GFlowNetscited by 13
  17. 2024Improved Off-policy Reinforcement Learning in Biological Sequence Designcited by 13
  18. 2024A Complexity-Based Theory of Compositionalitycited by 13
  19. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 12
  20. 2024Optimizing Backward Policies in GFlowNets via Trajectory Likelihood Maximizationcited by 12
  21. 2024Controllable Generation via Locally Constrained Resamplingcited by 11
  22. 2025BRiTE: Bootstrapping Reinforced Thinking Process to Enhance Language Model Reasoningcited by 10
  23. 2025Accurate and Diverse LLM Mathematical Reasoning via Automated PRM-Guided GFlowNetscited by 10
  24. 2024Latent Logic Tree Extraction for Event Sequence Explanation from LLMscited by 9
  25. 2024Beyond Squared Error: Exploring Loss Design for Enhanced Training of Generative Flow Networkscited by 9
  26. 2024Improving GFlowNets with Monte Carlo Tree Searchcited by 8
  27. 2025Revisiting Non-Acyclic GFlowNets in Discrete Environmentscited by 8
  28. 2025Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need?cited by 8
  29. 2026Mitigating Bias in Locally Constrained Decoding via Tractable Proposalscited by 8
  30. 2024Investigating Regularization of Self-Play Language Modelscited by 7
  31. 2024Guaranteed Generation from Large Language Modelscited by 7
  32. 2024GFlowNet Fine-tuning for Diverse Correct Solutions in Mathematical Reasoning Taskscited by 7
  33. 2025Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Predictioncited by 7
  34. 2026The Illusion of Stochasticity in LLMscited by 7
  35. 2024On Divergence Measures for Training GFlowNetscited by 6
  36. 2025Adaptive Destruction Processes for Diffusion Samplerscited by 6
  37. 2024Random Policy Evaluation Uncovers Policies of Generative Flow Networkscited by 5
  38. 2025Reinforced sequential Monte Carlo for amortised samplingcited by 5

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PowerFlow: Unlocking the Dual Nature of LLMs via Principled Distribution Matching
  2. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
  3. 2025Large Language Models as Discounted Bayesian Filters
  4. 2026Amortized Reasoning Tree Search: Decoupling Proposal and Decision in Large Language Models
  5. 2026Recipes for Steering and Scaling LLMs via Sampling
  6. 2024BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback
  7. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  8. 2026LLMs as High-Dimensional Nonlinear Autoregressive Models with Attention: Training, Alignment and Inference