mlsanityData to 14 Oct 2026

CM-Align: Consistency-based Multilingual Alignment for Large Language Models

arXiv 2509.08541 ↗First posted 10 Sep 2025cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Current large language models (LLMs) generally show a significant performance gap in alignment between English and other languages. To bridge this gap, existing research typically leverages the model's responses in English as a reference to select the best/worst responses in other languages, which are then used for Direct Preference Optimization (DPO) training. However, we argue that there are two limitations in the current methods that result in noisy multilingual preference data and further limited alignment performance: 1) Not all English responses are of high quality, and using a response with low quality may mislead the alignment for other languages. 2) Current methods usually use biased or heuristic approaches to construct multilingual preference pairs. To address these limitations, we design a consistency-based data selection method to construct high-quality multilingual preference data for improving multilingual alignment (CM-Align). Specifically, our method includes two parts: consistency-guided English reference selection and cross-lingual consistency-based multilingual preference data construction. Experimental results on three LLMs and three common tasks demonstrate the effectiveness and superiority of our method, which further indicates the necessity of constructing high-quality preference data.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 44 references that are in the map. First posted on arXiv 10 Sep 2025.

Builds on

30 of 44 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation4 Mar
  2. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization24 Feb
  3. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs2 Jul
  4. 2024Bootstrapping Language Models with DPO Implicit Rewards14 Jun
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  7. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  8. 2024MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization12 Jan
  9. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  10. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback29 Jul
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  8. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  9. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  10. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 479
  11. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 102
  12. 2024A Survey on Multilingual Large Language Models: Corpora, Alignment, and Biascited by 84
7 more
  1. 2024Language Imbalance Driven Rewarding for Multilingual Self-improvingcited by 28
  2. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  3. 2024Multilingual Knowledge Editing with Language-Agnostic Factual Neuronscited by 18
  4. 2025Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Expertscited by 12
  5. 2025CrowdSelect: Synthetic Instruction Data Selection with Multi-LLM Wisdomcited by 6
  6. 2023A Quality-based Syntactic Template Retriever for Syntactically-controlled Paraphrase Generationcited by 1
  7. 2024UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Datasetcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learningcited by 18
  2. 2026Adaptive Utilization of Low-Rank Adaptation via Conditioned Gatingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ComPO: Preference Alignment via Comparison Oracles
  2. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  3. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
  4. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2025Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment
  7. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  8. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization