mlsanityData to 7 Oct 2026

OpenAssistant Conversations -- Democratizing Large Language Model Alignment

arXiv 2304.07327 ↗First posted 14 Apr 2023cs.CL cs.AIcited by 712

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains. However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary. In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 complete and fully annotated conversation trees. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers. Models trained on OpenAssistant Conversations show consistent improvements on standard benchmarks over respective base models. We release our code and data under a fully permissive licence.

Key facts. Cited by 712 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 54 references that are in the map. First posted on arXiv 14 Apr 2023.

Builds on

23 of 54 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  2. 2022Training language models to follow instructions with human feedback4 Mar
  3. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  12. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
8 more
  1. 2023Instruction Tuning with GPT-4cited by 676
  2. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  3. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  4. 2020Artificial Intelligence, Values and Alignmentcited by 443
  5. 2022Teaching language models to support answers with verified quotescited by 290
  6. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  7. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  8. 2022The Effects of Data Quality on Machine Learning Performance on Tabular Datacited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 153; 2024: 293; 2025: 176; 2026: 90.

712 papers.

Sort:

By area:

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2023LIMA: Less Is More for Alignmentcited by 1,080
  6. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  7. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  10. 2024Self-Rewarding Language Modelscited by 654
  11. 2024A Survey on Large Language Models for Code Generationcited by 646
  12. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
38 more
  1. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  3. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  4. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  5. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  6. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  7. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  8. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  9. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  10. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  11. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  12. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  13. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  14. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  15. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 302
  16. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  17. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  18. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  19. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  20. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  21. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  22. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  23. 2023Challenges and Applications of Large Language Modelscited by 251
  24. 2023FireAct: Toward Language Agent Fine-tuningcited by 247
  25. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 237
  26. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  27. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  28. 2024Generative Representational Instruction Tuningcited by 227
  29. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  30. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  31. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  32. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  33. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  34. 2023OctoPack: Instruction Tuning Code Large Language Modelscited by 178
  35. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  36. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  37. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 170
  38. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 168

50 most cited of 712 All 712 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  2. 2023CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment
  3. 2026HAL: Inducing Human-likeness in LLMs with Alignment
  4. 2024Aligner: Efficient Alignment by Learning to Correct
  5. 2024Impact of Decoding Methods on Human Alignment of Conversational LLMs
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2025Aligning Spoken Dialogue Models from User Interactions
  8. 2023Large Language Model Alignment: A Survey