# Distributional Reinforcement Learning for Energy-Based Sequential Models

arXiv: https://arxiv.org/abs/1912.08517 · HTML: https://mlsanity.com/paper/1912.08517/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Dec 2019.

## Built on by (most cited first, 17 of 17)

- [Aligning Language Models with Preferences through f-divergence Minimization](https://mlsanity.com/paper/2302.08215/) (2023)
- [Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo](https://mlsanity.com/paper/2404.17546/) (2024)
- [On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting](https://mlsanity.com/paper/2206.00761/) (2022)
- [Controlling Conditional Language Models without Catastrophic Forgetting](https://mlsanity.com/paper/2112.00791/) (2021)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Energy-Based Models for Code Generation under Compilability Constraints](https://mlsanity.com/paper/2106.04985/) (2021)
- [Energy-Based Preference Model Offers Better Offline Alignment than the Bradley-Terry Preference Model](https://mlsanity.com/paper/2412.13862/) (2024)
- [Language Model Decoding as Direct Metrics Optimization](https://mlsanity.com/paper/2310.01041/) (2023)
- [BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback](https://mlsanity.com/paper/2402.02479/) (2024)
- [Guaranteed Generation from Large Language Models](https://mlsanity.com/paper/2410.06716/) (2024)
- [disco: a toolkit for Distributional Control of Generative Models](https://mlsanity.com/paper/2303.05431/) (2023)
- [Whatever Remains Must Be True: Filtering Drives Reasoning in LLMs, Shaping Diversity](https://mlsanity.com/paper/2512.05962/) (2025)
- [Sampling from Discrete Energy-Based Models with Quality/Efficiency Trade-offs](https://mlsanity.com/paper/2112.05702/) (2021)
- [Aligning language models with human preferences](https://mlsanity.com/paper/2404.12150/) (2024)
- [PIPA: Preference Alignment as Prior-Informed Statistical Estimation](https://mlsanity.com/paper/2502.05773/) (2025)
- [Structure-Aware Path Inference for Neural Finite State Transducers](https://mlsanity.com/paper/2312.13614/) (2023)
- [Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference](https://mlsanity.com/paper/2510.21184/) (2025)

## Abstract (arXiv)

Global Autoregressive Models (GAMs) are a recent proposal [Parshakova et al., CoNLL 2019] for exploiting global properties of sequences for data-efficient learning of seq2seq models. In the first phase of training, an Energy-Based model (EBM) over sequences is derived. This EBM has high representational power, but is unnormalized and cannot be directly exploited for sampling. To address this issue [Parshakova et al., CoNLL 2019] proposes a distillation technique, which can only be applied under limited conditions. By relating this problem to Policy Gradient techniques in RL, but in a distributional rather than optimization perspective, we propose a general approach applicable to any sequential EBM. Its effectiveness is illustrated on GAM-based experiments.