mlsanityData to 7 Oct 2026

MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Models

arXiv 2512.24693 ↗First posted 31 Dec 2025cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Preference alignment for recommenders and RAG

Abstract (arXiv)

Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human evaluation. Multi-turn reward models (RMs) offer a scalable alternative and can provide valuable signals for guiding LLM training. While recent work has advanced multi-turn textittraining techniques, effective automated textitevaluation specifically for multi-turn interactions lags behind. We observe that standard preference datasets, typically contrasting responses based only on the final conversational turn, provide insufficient signal to capture the nuances of multi-turn interactions. Instead, we find that incorporating contrasts spanning textitmultiple turns is critical for building robust multi-turn RMs. Motivated by this finding, we propose MUlti-Step Instruction Contrast (MUSIC), an unsupervised data augmentation strategy that synthesizes contrastive conversation pairs exhibiting differences across multiple turns. Leveraging MUSIC on the Skywork preference dataset, we train a multi-turn RM based on the Gemma-2-9B-Instruct model. Empirical results demonstrate that our MUSIC-augmented RM outperforms baseline methods, achieving higher alignment with judgments from advanced proprietary LLM judges on multi-turn conversations, crucially, without compromising performance on standard single-turn RM benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Preference alignment for recommenders and RAG, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 55 references that are in the map. First posted on arXiv 31 Dec 2025.

Builds on

44 of 55 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
32 more
  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  7. 2024Self-Rewarding Language Modelscited by 654
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  10. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  12. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  13. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 235
  14. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 232
  15. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  16. 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 171
  17. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  18. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  19. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  20. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  21. 2024Self-Taught Evaluatorscited by 70
  22. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  23. 2024Direct Multi-Turn Preference Optimization for Language Agentscited by 63
  24. 2024Aligning LLMs with Individual Preferences via Interactioncited by 63
  25. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  26. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 46
  27. 2025VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Datacited by 39
  28. 2025Consistently Simulating Human Personas with Multi-Turn Reinforcement Learningcited by 35
  29. 2024Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHFcited by 29
  30. 2025Dynamic and Generalizable Process Reward Modelingcited by 16
  31. 2025Aligning Large Language Models via Fully Self-Synthetic Datacited by 3
  32. 2025Iterative GRPO: Batch-Online Multi-Turn RL via Single-Turn RLHFcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Parrot: Enhancing Multi-Turn Instruction Following for Large Language Models
  2. 2026Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings
  3. 2026CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction
  4. 2026SDiaReward: Modeling and Benchmarking Spoken Dialogue Rewards with Modality and Colloquialness
  5. 2025InterMT: Multi-Turn Interleaved Preference Alignment with Human Feedback
  6. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment
  7. 2025A Survey on Multi-Turn Interaction Capabilities of Large Language Models
  8. 2025AU-Harness: An Open-Source Toolkit for Holistic Evaluation of Audio LLMs