mlsanityData to 14 Oct 2026

Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo

arXiv 2404.17546 ↗First posted 26 Apr 2024cs.LG cs.AI cs.CLcited by 90

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

also in 1 more
Abstract (arXiv)

Numerous capability and safety techniques of Large Language Models (LLMs), including RLHF, automated red-teaming, prompt engineering, and infilling, can be cast as sampling from an unnormalized target distribution defined by a given reward or potential function over the full sequence. In this work, we leverage the rich toolkit of Sequential Monte Carlo (SMC) for these probabilistic inference problems. In particular, we use learned twist functions to estimate the expected future value of the potential at each timestep, which enables us to focus inference-time computation on promising partial sequences. We propose a novel contrastive method for learning the twist functions, and establish connections with the rich literature of soft reinforcement learning. As a complementary application of our twisted SMC framework, we present methods for evaluating the accuracy of language model inference techniques using novel bidirectional SMC bounds on the log partition function. These bounds can be used to estimate the KL divergence between the inference and target distributions in both directions. We apply our inference evaluation techniques to show that twisted SMC is effective for sampling undesirable outputs from a pretrained model (a useful component of harmlessness training and automated red-teaming), generating reviews with varied sentiment, and performing infilling tasks.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 62 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

45 of 62 references.

References · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2022Red Teaming Language Models with Language Modelscited by 1,026
  12. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
33 more
  1. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  2. 2015Importance Weighted Autoencoderscited by 456
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  4. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 441
  5. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  6. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 376
  7. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 182
  8. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 161
  9. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  10. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  11. 2023Controlled Decoding from Language Modelscited by 135
  12. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 129
  13. 2020A Distributional Approach to Controlled Text Generationcited by 119
  14. 2022Language Model Cascadescited by 107
  15. 2023Amortizing intractable inference in large language modelscited by 97
  16. 2017Filtering Variational Objectivescited by 91
  17. 2023Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programscited by 80
  18. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  19. 2018Importance Weighting and Variational Inferencecited by 55
  20. 2023Training Chain-of-Thought via Latent-Variable Inferencecited by 47
  21. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 37
  22. 2015Neural Adaptive Sequential Monte Carlocited by 34
  23. 2021Learning to Give Checkable Answers with Prover-Verifier Gamescited by 32
  24. 2021Controlling Conditional Language Models without Catastrophic Forgettingcited by 32
  25. 2015Sandwiching the marginal likelihood using bidirectional Monte Carlocited by 20
  26. 2019Importance Weighted Hierarchical Variational Inferencecited by 20
  27. 2022Arithmetic Sampling: Parallel Diverse Decoding for Large Language Modelscited by 19
  28. 2022Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curvecited by 18
  29. 2023Long Horizon Temperature Scalingcited by 18
  30. 2019Distributional Reinforcement Learning for Energy-Based Sequential Modelscited by 17
  31. 2022Critic Sequential Monte Carlocited by 16
  32. 2023Improving Mutual Information Estimation with Annealed and Energy-Based Boundscited by 9
  33. 2016Measuring the reliability of MCMC inference with bidirectional Monte Carlocited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 35; 2026: 38.

90 papers.

Sort:

By area:

  1. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 224
  2. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  3. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 136
  4. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  5. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  6. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 94
  7. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 89
  8. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 88
  9. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 79
  10. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 49
  11. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 46
  12. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 39
38 more
  1. 2025Dynamic Search for Inference-Time Alignment in Diffusion Modelscited by 37
  2. 2024InfAlign: Inference-aware language model alignmentcited by 35
  3. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 32
  4. 2024Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNetscited by 32
  5. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  6. 2025Hypothesis-Driven Theory-of-Mind Reasoning for Large Language Modelscited by 29
  7. 2025Target Concrete Score Matching: A Holistic Framework for Discrete Diffusioncited by 25
  8. 2025DriftLite: Lightweight Drift Control for Inference-Time Scaling of Diffusion Modelscited by 23
  9. 2024Step-by-Step Reasoning for Math Problems via Twisted Sequential Monte Carlocited by 22
  10. 2025Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Samplingcited by 21
  11. 2025Rollout Roulette: A Probabilistic Inference Approach to Inference-Time Scaling of LLMs using Particle-Based Monte Carlo Methodscited by 20
  12. 2025Composition and Control with Distilled Energy Diffusion Models and Sequential Monte Carlocited by 20
  13. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 17
  14. 2024Estimating the Probabilities of Rare Outputs in Language Modelscited by 16
  15. 2025Qsharp: Provably Optimal Distributional RL for LLM Post-Trainingcited by 15
  16. 2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 14
  17. 2024Can a Bayesian Oracle Prevent Harm from an Agent?cited by 13
  18. 2025Self-Steering Language Modelscited by 13
  19. 2025Neurosymbolic Diffusion Modelscited by 13
  20. 2024A Critical Look At Tokenwise Reward-Guided Text Generationcited by 12
  21. 2025On the Query Complexity of Verifier-Assisted Language Generationcited by 12
  22. 2024Bayesian Optimization of Antibodies Informed by a Generative Model of Evolving Sequencescited by 11
  23. 2024Amortized Control of Continuous State Space Feynman-Kac Model for Irregular Time Seriescited by 10
  24. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 9
  25. 2025Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Predictioncited by 8
  26. 2025Steering Generative Models with Experimental Data for Protein Fitness Optimizationcited by 7
  27. 2025Inference-Time Scaling of Diffusion Language Models via Trajectory Refinementcited by 7
  28. 2026On the Power of (Approximate) Reward Models for Inference-Time Scalingcited by 7
  29. 2026Scaling Discovery through Test-Time Communicationcited by 7
  30. 2024Chemistry-Inspired Diffusion with Non-Differentiable Guidancecited by 6
  31. 2025Large (Vision) Language Models are Unsupervised In-Context Learnerscited by 6
  32. 2025Product of Experts for Visual Generationcited by 6
  33. 2025Reinforced sequential Monte Carlo for amortised samplingcited by 6
  34. 2026The Model Knows, the Decoder Finds: Future Value Guided Particle Power Samplingcited by 6
  35. 2025Towards Cost-Effective Reward Guided Text Generationcited by 5
  36. 2026Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillationcited by 5
  37. 2026Contrastive Distribution Matching for Amortized Sequential Monte Carlo in Discrete Diffusioncited by 5
  38. 2025Semantic Probabilistic Control of Language Modelscited by 4

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpening
  2. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  3. 2025Dynamic Vocabulary Pruning: Stable LLM-RL by Taming the Tail
  4. 2026Power-SMC: Low-Latency Sequence-Level Power Sampling for Training-Free LLM Reasoning
  5. 2025On the Limits of Test-Time Compute: Sequential Reward Filtering for Better Inference
  6. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models
  7. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learning
  8. 2025Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference