mlsanityData to 7 Oct 2026

OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

arXiv 2309.11235 ↗First posted 20 Sep 2023cs.CLcited by 250

Not on the map. Why?

Cited by 42 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Nowadays, open-source large language models like LLaMA have emerged. Recent developments have incorporated supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to align these models with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of a small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the C(onditioned)-RLFT, which regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. Through extensive experiments on three standard benchmarks, our openchat-13b fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. Moreover, we use AGIEval to validate the model generalization performance, in which only openchat-13b surpasses the base model. Finally, we conduct a series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at https://github.com/imoneoi/openchat and https://huggingface.co/openchat.

Key facts. Cited by 250 papers in the map (data through 7 Oct 2026). Builds on 31 of its 41 references that are in the map. First posted on arXiv 20 Sep 2023.

Builds on

31 of 41 references.

References · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
19 more
  1. 2023A Survey of Large Language Modelscited by 2,537
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  5. 2023LIMA: Less Is More for Alignmentcited by 1,080
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  7. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  8. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  10. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  11. 2023Instruction Tuning with GPT-4cited by 676
  12. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  13. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  14. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  15. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  16. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  17. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  18. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  19. 2020Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Trainingcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 142; 2025: 71; 2026: 18.

250 papers.

Sort:

By area:

  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  5. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  6. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 319
  7. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  8. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  9. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  10. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  11. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  12. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
38 more
  1. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  2. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 222
  3. 2024ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Modelscited by 204
  4. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  5. 2023InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviewscited by 177
  6. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 125
  7. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  8. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectivenesscited by 117
  9. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  10. 2024On Prompt-Driven Safeguarding for Large Language Modelscited by 109
  11. 2024LLM Agents can Autonomously Hack Websitescited by 108
  12. 2023Generative Language Models Exhibit Social Identity Biasescited by 107
  13. 2023MoDS: Model-oriented Data Selection for Instruction Tuningcited by 107
  14. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 107
  15. 2024Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignmentcited by 104
  16. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  17. 2024FOFO: A Benchmark to Evaluate LLMs' Format-Following Capabilitycited by 79
  18. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  19. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  20. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 68
  21. 2024ChatMusician: Understanding and Generating Music Intrinsically with LLMcited by 67
  22. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
  23. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  24. 2024We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMscited by 61
  25. 2023EgoThink: Evaluating First-Person Perspective Thinking Capability of Vision-Language Modelscited by 59
  26. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  27. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  28. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  29. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  30. 2025A Survey on the Optimization of Large Language Model-based Agentscited by 45
  31. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  32. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 38
  33. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  34. 2024Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languagescited by 36
  35. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  36. 2024AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queriescited by 32
  37. 2023Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoningcited by 30
  38. 2024VidEgoThink: Assessing Egocentric Video Understanding Capabilities for Embodied AIcited by 30

50 most cited of 250 All 250 (JSON)