mlsanityData to 14 Oct 2026

SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtracking

arXiv 2306.05426 ↗First posted 8 Jun 2023cs.LG cs.AIcited by 13

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

In many domains, autoregressive models can attain high likelihood on the task of predicting the next observation. However, this maximum-likelihood (MLE) objective does not necessarily match a downstream use-case of autoregressively generating high-quality sequences. The MLE objective weights sequences proportionally to their frequency under the data distribution, with no guidance for the model's behaviour out of distribution (OOD): leading to compounding error during autoregressive generation. In order to address this compounding error problem, we formulate sequence generation as an imitation learning (IL) problem. This allows us to minimize a variety of divergences between the distribution of sequences generated by an autoregressive model and sequences from a dataset, including divergences with weight on OOD generated sequences. The IL framework also allows us to incorporate backtracking by introducing a backspace action into the generation process. This further mitigates the compounding error problem by allowing the model to revert a sampled token if it takes the sequence OOD. Our resulting method, SequenceMatch, can be implemented without adversarial training or architectural changes. We identify the SequenceMatch-chi2 divergence as a more suitable training objective for autoregressive models which are used for generation. We show that empirically, SequenceMatch training leads to improvements over MLE on text generation with language models and arithmetic.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 43 references that are in the map. First posted on arXiv 8 Jun 2023.

Builds on

28 of 43 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021IQ-Learn: Inverse soft-Q Learning for Imitation23 Jun
  2. 2020Adversarial Soft Advantage Fitting: Imitation Learning without Policy Optimization23 Jun
  3. 2019Imitation Learning via Off-Policy Distribution Matching10 Dec

Other references · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  4. 2022Generative Adversarial Networkscited by 5,627
  5. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2016Generative Adversarial Imitation Learningcited by 1,597
13 more
  1. 2018Hierarchical Neural Story Generationcited by 1,193
  2. 2019Neural Text Generation with Unlikelihood Trainingcited by 503
  3. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 335
  4. 2019Scaling Autoregressive Video Modelscited by 190
  5. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 108
  6. 2020A Survey on Generative Adversarial Networks: Variants, Applications, and Trainingcited by 97
  7. 2020A Theoretical Analysis of the Repetition Problem in Text Generationcited by 83
  8. 2018Toward Diverse Text Generation with Inverse Reinforcement Learningcited by 41
  9. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 36
  10. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 29
  11. 2022A Simple Contrastive Learning Objective for Alleviating Neural Text Degenerationcited by 18
  12. 2020Lessons Learned from the Training of GANs on Artificial Datasetscited by 5
  13. 2021Training Wasserstein GANs without gradient penaltiescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 7; 2026: 2.

13 papers.

Sort:

By area:

  1. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  2. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  3. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 41
  4. 2024Hybrid Inverse Reinforcement Learningcited by 32
  5. 2024Backtracking Improves Generation Safetycited by 32
  6. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  7. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
  8. 2025Structured Agent Distillation for Large Language Modelcited by 13
  9. 2025Learning Safety Constraints for Large Language Modelscited by 13
  10. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 12
  11. 2025ERGO: Entropy-guided Resetting for Generation Optimization in Multi-turn Language Modelscited by 6
  12. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
1 more
  1. 2026The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Workscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unifying Autoregressive and Diffusion-Based Sequence Generation
  2. 2025Computational-Statistical Tradeoffs at the Next-Token Prediction Barrier: Autoregressive and Imitation Learning under Misspecification
  3. 2020Autoregressive Knowledge Distillation through Imitation Learning
  4. 2025Ideas in Inference-time Scaling can Benefit Generative Pre-training Algorithms
  5. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation
  6. 2026Stochastic Autoregressive Learning
  7. 2025TIMED: Adversarial and Autoregressive Refinement of Diffusion-Based Time Series Generation
  8. 2025Fast Autoregressive Models for Continuous Latent Generation