mlsanityData to 7 Oct 2026

PHOENIX: Open-Source Language Adaption for Direct Preference Optimization

arXiv 2401.10580 ↗First posted 19 Jan 2024cs.CLcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models have gained immense importance in recent years and have demonstrated outstanding results in solving various tasks. However, despite these achievements, many questions remain unanswered in the context of large language models. Besides the optimal use of the models for inference and the alignment of the results to the desired specifications, the transfer of models to other languages is still an underdeveloped area of research. The recent publication of models such as Llama-2 and Zephyr has provided new insights into architectural improvements and the use of human feedback. However, insights into adapting these techniques to other languages remain scarce. In this paper, we build on latest improvements and apply the Direct Preference Optimization(DPO) approach to the German language. The model is available at https://huggingface.co/DRXD1000/Phoenix.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 27 of its 36 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

27 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  2. 2023LIMA: Less Is More for Alignment18 May
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 23

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Mistral 7Bcited by 3,065
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
11 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  5. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  6. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  8. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  9. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  10. 2023A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Modelscited by 88
  11. 2023Efficient Language Model Training through Cross-Lingual and Progressive Transfer Learningcited by 26

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?cited by 10

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Direct Preference Optimization
  2. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  3. 2024Multi-Reference Preference Optimization for Large Language Models
  4. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  5. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  6. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  7. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  8. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization