mlsanityData to 7 Oct 2026

Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control

arXiv 1611.02796 ↗First posted 9 Nov 2016cs.LG cs.AIcited by 100

Not on the map. Why?

Cited by 46 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper proposes a general method for improving the structure and quality of sequences generated by a recurrent neural network (RNN), while maintaining information originally learned from data, as well as sample diversity. An RNN is first pre-trained on data using maximum likelihood estimation (MLE), and the probability distribution over the next token in the sequence learned by this model is treated as a prior policy. Another RNN is then trained using reinforcement learning (RL) to generate higher-quality outputs that account for domain-specific incentives while retaining proximity to the prior policy of the MLE RNN. To formalize this objective, we derive novel off-policy RL methods for RNNs from KL-control. The effectiveness of the approach is demonstrated on two applications; 1) generating novel musical melodies, and 2) computational molecular generation. For both problems, we show that the proposed method improves the desired properties and structure of the generated sequences, while maintaining information learned from data.

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Nov 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 3; 2020: 3; 2021: 7; 2022: 7; 2023: 15; 2024: 16; 2025: 30; 2026: 19.

100 papers.

Sort:

By area:

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2022Red Teaming Language Models with Language Modelscited by 983
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  7. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  8. 2022Teaching language models to support answers with verified quotescited by 290
  9. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  11. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  12. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
38 more
  1. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  2. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  3. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  4. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  5. 2019Benchmarking Batch Deep Reinforcement Learning Algorithmscited by 108
  6. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  7. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  8. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  9. 2023Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedbackcited by 71
  10. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  11. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  12. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  13. 2024IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologuescited by 40
  14. 2024Moral Alignment for LLM Agentscited by 39
  15. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 35
  16. 2023Language Model Alignment with Elastic Resetcited by 34
  17. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  18. 2024MusicRL: Aligning Music Generation to Human Preferencescited by 32
  19. 2025GVPO: Group Variance Policy Optimization for Large Language Model Post-Trainingcited by 32
  20. 2025Enhancing Personalized Multi-Turn Dialogue with Curiosity Rewardcited by 25
  21. 2021Creativity and Machine Learning: A Surveycited by 21
  22. 2024Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modelingcited by 20
  23. 2023Mitigating Open-Vocabulary Caption Hallucinationscited by 18
  24. 2025Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in Its Latent Thoughtscited by 18
  25. 2025A Comedy of Estimators: On KL Regularization in RL Training of LLMscited by 18
  26. 2021Policy Information Capacity: Information-Theoretic Measure for Task Complexity in Deep Reinforcement Learningcited by 15
  27. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  28. 2024SILMM: Self-Improving Large Multimodal Models for Compositional Text-to-Image Generationcited by 15
  29. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  30. 2025In-Training Defenses against Emergent Misalignment in Language Modelscited by 14
  31. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  32. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12
  33. 2023Reinforcement Learning for Generative AI: A Surveycited by 11
  34. 2025Reinforcement Fine-Tuning for Materials Designcited by 11
  35. 2023Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challengescited by 10
  36. 2024Diversity-Rewarded CFG Distillationcited by 9
  37. 2025Lightweight and Direct Document Relevance Optimization for Generative Information Retrievalcited by 9
  38. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspectivecited by 9

50 most cited of 100 All 100 (JSON)