mlsanityData to 14 Oct 2026

Sailor: Open Language Models for South-East Asia

arXiv 2404.03608 ↗First posted 4 Apr 2024cs.CL cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

We present Sailor, a family of open language models ranging from 0.5B to 7B parameters, tailored for South-East Asian (SEA) languages. These models are continually pre-trained from Qwen1.5, a great language model for multilingual use cases. From Qwen1.5, Sailor models accept 200B to 400B tokens, primarily covering the languages of English, Chinese, Vietnamese, Thai, Indonesian, Malay, and Lao. The training leverages several techniques, including BPE dropout for improving the model robustness, aggressive data cleaning and deduplication, and small proxy models to optimize data mixture. Experimental results on four typical tasks indicate that Sailor models demonstrate strong performance across different benchmarks, including commonsense reasoning, question answering, reading comprehension and examination. Embracing the open-source spirit, we share our insights through this report to spark a wider interest in developing large language models for multilingual use cases.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 40 of its 64 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

40 of 64 references.

References · 40

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
28 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2020GLU Variants Improve Transformercited by 1,804
  3. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  4. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  5. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  6. 2024Yi: Open Foundation Models by 01.AIcited by 719
  7. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  9. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  10. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  11. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 548
  12. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  13. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  14. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  15. 2019BPE-Dropout: Simple and Effective Subword Regularizationcited by 195
  16. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  17. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  18. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  19. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  20. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 110
  21. 2023SeaLLMs -- Large Language Models for Southeast Asiacited by 104
  22. 2024LLaMA Beyond English: An Empirical Study on Language Capability Transfercited by 95
  23. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 72
  24. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
  25. 2023Typhoon: Thai Large Language Modelscited by 30
  26. 2024Weaver: Foundation Models for Creative Writingcited by 27
  27. 2024Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Modelscited by 15
  28. 2023VinaLLaMA: LLaMA-based Vietnamese Foundation Modelcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 6; 2026: 4.

19 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  2. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 43
  3. 2024Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scalecited by 37
  4. 2025Scaling Embedding Layers in Language Modelscited by 20
  5. 2025RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programscited by 17
  6. 2024Typhoon 2: A Family of Open Text and Multimodal Thai Large Language Modelscited by 12
  7. 2024Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglementcited by 11
  8. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 8
  9. 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insightscited by 7
  10. 2025DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detectioncited by 5
  11. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  12. 2024SailCompass: Towards Reproducible and Robust Evaluation for Southeast Asian Languagescited by 3
7 more
  1. 2024A Comparative Study of LLM-based ASR and Whisper in Low Resource and Code Switching Scenariocited by 1
  2. 2026UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editingcited by 1
  3. 2026In-Place Tokenizer Expansion for Pre-trained LLMscited by 1
  4. 2024SnakModel: Lessons Learned from Training an Open Danish Large Language Modelcited by 0
  5. 2025A Benchmark Dataset and Evaluation Framework for Vietnamese Large Language Models in Customer Supportcited by 0
  6. 2025Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Modelscited by 0
  7. 2026TSWAP: A Multilingual Retrieval-Augmented Thai Wellness Advisorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models
  2. 2025LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
  3. 2023YAYI 2: Multilingual Open-Source Large Language Models
  4. 2026Typhoon-S: Minimal Open Post-Training for Sovereign Large Language Models
  5. 2025KORMo: Korean Open Reasoning Model for Everyone
  6. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers
  7. 2025Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
  8. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation