mlsanityData to 14 Oct 2026

Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages

arXiv 2512.08777 ↗First posted 9 Dec 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We propose a post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languages lack both datasets written by native speakers and instruction-tuned language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligned language model without any instruction-tuning data in the target language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine-translated data and multilingual finetuning. We conduct a case study on Norwegian Bokmaal and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 81 references that are in the map. First posted on arXiv 9 Dec 2025.

Builds on

48 of 81 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs2 Jul
  2. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback29 Jul
  3. 2023LIMA: Less Is More for Alignment18 May
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  10. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  12. 2024Self-Rewarding Language Modelscited by 681
32 more
  1. 2019How multilingual is Multilingual BERT?cited by 645
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  3. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  4. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  5. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 295
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  7. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  8. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  9. 2017Unsupervised Neural Machine Translationcited by 149
  10. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  11. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  12. 2013The Optimal Reward Baseline for Gradient-Based Reinforcement Learningcited by 85
  13. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  14. 2022Democratizing Neural Machine Translation with OPUS-MTcited by 58
  15. 2024ALLaM: Large Language Models for Arabic and Englishcited by 55
  16. 2019Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translationcited by 45
  17. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
  18. 2023Typhoon: Thai Large Language Modelscited by 30
  19. 2025Seed-X: Building Strong Multilingual Translation LLM with 7B Parameterscited by 21
  20. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  21. 2017NoReC: The Norwegian Review Corpuscited by 17
  22. 2022Understanding Translationese in Cross-Lingual Summarizationcited by 14
  23. 2023Camoscio: an Italian Instruction-tuned LLaMAcited by 13
  24. 2024Small Languages, Big Models: A Study of Continual Training on Languages of Norwaycited by 12
  25. 2025Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basquecited by 12
  26. 2024The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspectivecited by 8
  27. 2025NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmarkcited by 8
  28. 2023NorQuAD: Norwegian Question Answering Datasetcited by 7
  29. 2023From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Modelscited by 5
  30. 2024LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoningcited by 4
  31. 2025Got Compute, but No Data: Lessons From Post-training a Finnish LLMcited by 3
  32. 2025Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Studycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  2. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques
  3. 2026Language Chain in Alignment: Cross-lingual Ranking Preference Optimization
  4. 2024Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment
  5. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
  6. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  7. 2023Aligning Language Models with Offline Learning from Human Feedback
  8. 2024LIONs: An Empirically Optimized Approach to Align Language Models