mlsanityData to 14 Oct 2026

MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining

arXiv 2507.01785 ↗First posted 2 Jul 2025cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Data quality is a critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English. We introduce MuRating, a scalable framework that transfers high-quality English data-quality signals into a single rater for 17 target languages. MuRating aggregates multiple English "raters" via pairwise comparisons to learn unified document-quality scores,then projects these judgments through translation to train a multilingual evaluator on monolingual, cross-lingual, and parallel text pairs. Applied to web data, MuRating selects balanced subsets of English and multilingual content to pretrain a 1.2 B-parameter LLaMA model. Compared to strong baselines, including QuRater, AskLLM, DCLM and so on, our approach boosts average accuracy on both English benchmarks and multilingual evaluations, with especially large gains on knowledge-intensive tasks. We further analyze translation fidelity, selection biases, and underrepresentation of narrative material, outlining directions for future work.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 58 of its 70 references that are in the map. First posted on arXiv 2 Jul 2025.

Builds on

58 of 70 references.

References · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
38 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  9. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  10. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  11. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  12. 2023Textbooks Are All You Needcited by 569
  13. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  14. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  15. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  16. 2023Scaling Data-Constrained Language Modelscited by 403
  17. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 399
  18. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  19. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  20. 2023Data Selection for Language Models via Importance Resamplingcited by 329
  21. 2023From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuningcited by 307
  22. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 303
  23. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  24. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  25. 2024QuRating: Selecting High-Quality Data for Training Language Modelscited by 157
  26. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  27. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  28. 2024Superfiltering: Weak-to-Strong Data Filtering for Fast Instruction-Tuningcited by 138
  29. 2024When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboardscited by 137
  30. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 118
  31. 2024How to Train Data-Efficient LLMscited by 93
  32. 2023To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisiscited by 92
  33. 2017Learning how to Active Learn: A Deep Reinforcement Learning Approachcited by 79
  34. 2023SlimPajama-DC: Understanding Data Combinations for LLM Trainingcited by 72
  35. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  36. 2024Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performancecited by 61
  37. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57
  38. 2024A New Massive Multilingual Dataset for High-Performance Language Technologiescited by 56

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2026"UberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Datasetcited by 3
  2. 2026DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Modelscited by 1
  3. 2025Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretrainingcited by 0
  4. 2026Fisher-Guided Submodular Data Selection for Continual Pre-Training of Large Language Modelscited by 0
  5. 2026Adapting English Quality Classifiers for Multilingual LLM Pretraining Data Selectioncited by 0