mlsanityData to 14 Oct 2026

MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages

arXiv 2509.26601 ↗First posted 30 Sep 2025cs.CL cs.AI cs.LGcited by 3

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Ensuring native-like quality of large language model (LLM) responses across many languages is challenging. To address this, we introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms. Using MENLO, we create a dataset of 6,423 human-annotated prompt-response preference pairs covering four quality dimensions with high inter-annotator agreement in 47 language varieties. Our evaluation reveals that zero-shot LLM judges benefit significantly from pairwise evaluation and our structured annotation rubrics, yet they still underperform human annotators on our dataset. We demonstrate substantial improvements through fine-tuning with reinforcement learning, reward shaping, and multi-task learning approaches. Additionally, we show that RL-trained judges can serve as generative reward models to enhance LLMs' multilingual proficiency, though discrepancies with human judgment remain. Our findings suggest promising directions for scalable multilingual evaluation and preference alignment. We release our dataset and evaluation framework to support further research in multilingual LLM evaluation (https://huggingface.co/datasets/facebook/menlo).

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 32 of its 39 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

32 of 39 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models23 Oct
  2. 2024A Roadmap to Pluralistic Alignment7 Feb

Other references · 30

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  3. 2023PaLM 2 Technical Reportcited by 1,162
  4. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  5. 2023MEGA: Multilingual Evaluation of Generative AIcited by 293
  6. 2022Challenges and Strategies in Cross-Cultural NLPcited by 218
  7. 2024BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languagescited by 173
  8. 2021XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluationcited by 168
  9. 2022Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languagescited by 102
  10. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  11. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  12. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
18 more
  1. 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalizationcited by 60
  2. 2023MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Taskscited by 57
  3. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
  4. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settingscited by 50
  5. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  6. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  7. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
  8. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  9. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 26
  10. 2024Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMscited by 19
  11. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 18
  12. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  13. 2025MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMscited by 16
  14. 2016Crowd-sourcing NLG Data: Pictures Elicit Better Datacited by 11
  15. 2021Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERTcited by 5
  16. 2024OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Modelscited by 4
  17. 2024Evaluating Large Language Models with Tests of Spanish as a Foreign Language: Pass or Fail?cited by 3
  18. 2025Testing Low-Resource Language Support in LLMs Using Language Proficiency Exams: the Case of Luxembourgishcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  2. 2026K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contextscited by 3
  3. 2026τ-Multilingual: Benchmarking Voice Agents Across Languagescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Do
  2. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
  3. 2023On Diversified Preferences of Large Language Model Alignment
  4. 2025Evaluating the Diversity and Quality of LLM Generated Content
  5. 2026Meta-Learning Preferences for Multilingual LLM Alignment
  6. 2026Steerable Cultural Preference Optimization of Reward Models
  7. 2026CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
  8. 2024LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback