Incoherence in Goal-Conditioned Autoregressive Models
Not on the map. Why?
Abstract (arXiv)
We investigate mathematically the notion of incoherence: a structural issue with reinforcement learning policies derived by naive goal-conditioning of autoregressive models. We focus on the process of re-training models on their own actions, that is, fine-tuning offline-learned policies with online RL. We prove that it decreases incoherence and leads to an improvement in return, and we aim to characterize the resulting trajectory of policies. By re-framing standard notions of control-as-inference and soft Q learning, we establish a three-way correspondence with two other ways of understanding the iterative re-training process: as folding the posterior into the reward and, in the deterministic case, as decreasing the temperature parameter; the correspondence has computational content via the training-inference trade-off. Through soft-conditioning generative models, we discuss the link between incoherence and the effective horizon.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 8 Oct 2025.
Builds on
34 of 46 references.
References · 34
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2020Learning to summarize from human feedbackcited by 2,562
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
- 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
- 2023Large Language Models as Optimizerscited by 830
22 more
- 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
- 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
- 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
- 2018Maximum a Posteriori Policy Optimisationcited by 318
- 2023Role-Play with Large Language Modelscited by 290
- 2023Auto-GPT for Online Decision Making: Benchmarks and Additional Opinionscited by 250
- 2017Thinking Fast and Slow with Deep Learning and Tree Searchcited by 245
- 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 175
- 2022Language Models as Agent Modelscited by 131
- 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
- 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
- 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
- 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
- 2020Residual Energy-Based Models for Text Generationcited by 95
- 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
- 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
- 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31
- 2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
- 2022A Primer on Maximum Causal Entropy Inverse Reinforcement Learningcited by 13
- 2023Conditioning Predictive Models: Risks and Strategiescited by 11
- 2022Upside-Down Reinforcement Learning Can Diverge in Stochastic Environments With Episodic Resetscited by 10
- 2024Limitations of Agents Simulated by Predictive Modelscited by 1
Built on by
Not cited yet.