mlsanityData to 7 Oct 2026

TS-Align: A Teacher-Student Collaborative Framework for Scalable Iterative Finetuning of Large Language Models

arXiv 2405.20215 ↗First posted 30 May 2024cs.CLcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Mainstream approaches to aligning large language models (LLMs) heavily rely on human preference data, particularly when models require periodic updates. The standard process for iterative alignment of LLMs involves collecting new human feedback for each update. However, the data collection process is costly and challenging to scale. To address this issue, we introduce the "TS-Align" framework, which fine-tunes a policy model using pairwise feedback data automatically mined from its outputs. This automatic mining process is efficiently accomplished through the collaboration between a large-scale teacher model and a small-scale student model. The policy fine-tuning process can be iteratively repeated using on-policy generations within our proposed teacher-student collaborative framework. Through extensive experiments, we demonstrate that our final aligned policy outperforms the base policy model with an average win rate of 69.7% across seven conversational or instruction-following datasets. Furthermore, we show that the ranking capability of the teacher is effectively distilled into the student through our pipeline, resulting in a small-scale yet effective reward model for policy model alignment.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 76 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

41 of 76 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  2. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss27 Dec
  3. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets20 Jul
  4. 2023Preference Ranking Optimization for Human Alignment30 Jun
  5. 2023Aligning Large Language Models through Synthetic Feedback23 May
  6. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 34

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2020Learning to summarize from human feedbackcited by 2,562
22 more
  1. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  13. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  14. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  15. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  16. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 325
  17. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  18. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  19. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  20. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  21. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Modelscited by 27
  22. 2021MDD-Eval: Self-Training on Augmented Data for Multi-Domain Dialogue Evaluationcited by 13

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  2. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
  3. 2025It Helps to Take a Second Opinion: Teaching Smaller LLMs to Deliberate Mutually via Selective Rationale Optimisationcited by 1
  4. 2024Hybrid Student-Teacher Large Language Model Refinement for Cancer Toxicity Symptom Extractioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Pedagogical Alignment of Large Language Models
  2. 2024Aligning Large Language Models by On-Policy Self-Judgment
  3. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Models
  4. 2024Aligning Large Language Models via Fine-grained Supervision
  5. 2025Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
  6. 2024Aligning Teacher with Student Preferences for Tailored Training Data Generation
  7. 2024LIONs: An Empirically Optimized Approach to Align Language Models
  8. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models