mlsanityData to 7 Oct 2026

Baichuan Alignment Technical Report

arXiv 2410.14940 ↗First posted 19 Oct 2024cs.LG cs.CLcited by 6

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We introduce Baichuan Alignment, a detailed analysis of the alignment techniques employed in the Baichuan series of models. This represents the industry's first comprehensive account of alignment methodologies, offering valuable insights for advancing AI research. We investigate the critical components that enhance model performance during the alignment process, including optimization methods, data strategies, capability enhancements, and evaluation processes. The process spans three key stages: Prompt Augmentation System(PAS), Supervised Fine-Tuning(SFT), and Preference Alignment. The problems encountered, the solutions applied, and the improvements made are thoroughly recorded. Through comparisons across well-established benchmarks, we highlight the technological advancements enabled by Baichuan Alignment. Baichuan-Instruct is an internal model, while Qwen2-Nova-72B and Llama3-PBM-Nova-70B are instruct versions of the Qwen2-72B and Llama-3-70B base models, optimized through Baichuan Alignment. Baichuan-Instruct demonstrates significant improvements in core capabilities, with user experience gains ranging from 17% to 28%, and performs exceptionally well on specialized benchmarks. In open-source benchmark evaluations, both Qwen2-Nova-72B and Llama3-PBM-Nova-70B consistently outperform their respective official instruct versions across nearly all datasets. This report aims to clarify the key technologies behind the alignment process, fostering a deeper understanding within the community. Llama3-PBM-Nova-70B model is available at https://huggingface.co/PKU-Baichuan-MLSystemLab/Llama3-PBM-Nova-70B.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 83 of its 96 references that are in the map. First posted on arXiv 19 Oct 2024.

Builds on

83 of 96 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models19 Jun
  2. 2024SoFA: Shielded On-the-fly Alignment via Priority Rule Following27 Feb
  3. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning25 Dec
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 79

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  9. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  10. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  11. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,053
38 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  8. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  13. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  14. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  15. 2024Qwen2 Technical Reportcited by 2,076
  16. 2024Mixtral of Expertscited by 1,785
  17. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  18. 2023PaLM 2 Technical Reportcited by 1,154
  19. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  20. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  21. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  22. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  23. 2022Editing Models with Task Arithmeticcited by 1,076
  24. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  25. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  26. 2024Yi: Open Foundation Models by 01.AIcited by 706
  27. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  28. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  29. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  30. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  31. 2024Qwen2-Audio Technical Reportcited by 543
  32. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  33. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  34. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  35. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 301
  36. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  37. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  38. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learningcited by 152
  2. 2025Baichuan-M2: Scaling Medical Capability with Large Verifier Systemcited by 55
  3. 2026Baichuan-M3: Modeling Clinical Inquiry for Reliable Medical Decision-Makingcited by 10
  4. 2025System Message Generation for User Preferences using Open-Source Modelscited by 1
  5. 2025Benchmarking Chinese Commonsense Reasoning with a Multi-hop Reasoning Perspectivecited by 1
  6. 2025KunLunBaizeRAG: Reinforcement Learning Driven Inference Performance Leap for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Models
  2. 2025RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models
  3. 2025Towards Efficient and Effective Alignment of Large Language Models
  4. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment
  5. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  6. 2025PIKA: Expert-Level Synthetic Datasets for Post-Training Alignment from Scratch
  7. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
  8. 2024Is Free Self-Alignment Possible?