mlsanityData to 7 Oct 2026

A Framework for Fine-Tuning LLMs using Heterogeneous Feedback

arXiv 2408.02861 ↗First posted 5 Aug 2024cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models (LLMs) have been applied to a wide range of tasks, including text summarization, web navigation, and chatbots. They have benefitted from supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) following an unsupervised pretraining. These datasets can be difficult to collect, limited in scope, and vary in sample quality. Additionally, datasets can vary extensively in supervision format, from numerical to binary as well as multi-dimensional with many different values. We present a framework for fine-tuning LLMs using heterogeneous feedback, which has two main components. First, we combine the heterogeneous feedback data into a single supervision format, compatible with methods like SFT and RLHF. Next, given this unified feedback dataset, we extract a high-quality and diverse subset to obtain performance increases potentially exceeding the full dataset. We conduct extensive experiments to understand the effectiveness of these techniques for incorporating heterogeneous feedback, and demonstrate improvements from using a high-quality and diverse subset of the data. We find that our framework is able to improve models in multiple areas simultaneously, such as in instruction following and bias reduction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 19 references that are in the map. First posted on arXiv 5 Aug 2024.

Builds on

14 of 19 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2018Gender Bias in Coreference Resolution: Evaluation and Debiasing Methodscited by 657
  8. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  9. 2023Platypus: Quick, Cheap, and Powerful Refinement of LLMscited by 150
  10. 2024A Survey on Data Selection for LLM Instruction Tuningcited by 76
  11. 2023Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Taskscited by 53
  12. 2023Self-Evolved Diverse Data Sampling for Efficient Instruction Tuningcited by 52

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunities
  2. 2024Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF
  3. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning
  4. 2024Aligning Large Language Models via Fine-grained Supervision
  5. 2024A critical review of methods and challenges in large language models
  6. 2025Evaluating the Diversity and Quality of LLM Generated Content
  7. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexity
  8. 2026FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents