# Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

arXiv: https://arxiv.org/abs/2502.12982 · HTML: https://mlsanity.com/paper/2502.12982/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 74 of its 88 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Builds on (most cited first, 30 of 74)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints](https://mlsanity.com/paper/2305.13245/) (2023)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Tulu 3: Pushing Frontiers in Open Language Model Post-Training](https://mlsanity.com/paper/2411.15124/) (2024)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [A General Theoretical Paradigm to Understand Learning from Human Preferences](https://mlsanity.com/paper/2310.12036/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies](https://mlsanity.com/paper/2404.06395/) (2024)
- [Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM](https://mlsanity.com/paper/2104.04473/) (2021)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models](https://mlsanity.com/paper/2401.01335/) (2024)
- [ByT5: Towards a token-free future with pre-trained byte-to-byte models](https://mlsanity.com/paper/2105.13626/) (2021)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning](https://mlsanity.com/paper/2310.06694/) (2023)
- [DataComp-LM: In search of the next generation of training sets for language models](https://mlsanity.com/paper/2406.11794/) (2024)
- [XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning](https://mlsanity.com/paper/2005.00333/) (2020)
- [Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs](https://mlsanity.com/paper/2410.18451/) (2024)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://mlsanity.com/paper/2402.07827/) (2024)
- [Direct Language Model Alignment from Online AI Feedback](https://mlsanity.com/paper/2402.04792/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [Disentangling Length from Quality in Direct Preference Optimization](https://mlsanity.com/paper/2403.19159/) (2024)
- [Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation](https://mlsanity.com/paper/2412.03304/) (2024)

## Built on by (most cited first, 18 of 18)

- [When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners](https://mlsanity.com/paper/2505.15257/) (2025)
- [MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages](https://mlsanity.com/paper/2506.19468/) (2025)
- [Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures](https://mlsanity.com/paper/2510.24081/) (2025)
- [SeaLLMs-Audio: Large Audio-Language Models for Southeast Asia](https://mlsanity.com/paper/2511.01670/) (2025)
- [Batayan: A Filipino NLP benchmark for evaluating Large Language Models](https://mlsanity.com/paper/2502.14911/) (2025)
- [Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling](https://mlsanity.com/paper/2604.25578/) (2026)
- [Compass-V2 Technical Report](https://mlsanity.com/paper/2504.15527/) (2025)
- [XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity](https://mlsanity.com/paper/2605.05662/) (2026)
- [VietBinoculars: A Zero-Shot Approach for Detecting Vietnamese LLM-Generated Text](https://mlsanity.com/paper/2509.26189/) (2025)
- [LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoning](https://mlsanity.com/paper/2510.09189/) (2025)
- [BURMESE-SAN: Burmese NLP Benchmark for Evaluating Large Language Models](https://mlsanity.com/paper/2602.18788/) (2026)
- [VietJobs: A Vietnamese Job Advertisement Dataset](https://mlsanity.com/paper/2603.05262/) (2026)
- [In-Place Tokenizer Expansion for Pre-trained LLMs](https://mlsanity.com/paper/2607.15232/) (2026)
- [BYOL: Bring Your Own Language Into LLMs](https://mlsanity.com/paper/2601.10804/) (2026)
- [Orchestrating Heterogeneous Experts: A Scalable MoE Framework with Anisotropy-Preserving Fusion](https://mlsanity.com/paper/2602.00003/) (2025)
- [Phoenix-VL 1.5 Medium Technical Report](https://mlsanity.com/paper/2605.10391/) (2026)
- [A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering](https://mlsanity.com/paper/2605.22099/) (2026)
- [IndoBias: A Dual Track Culturally Grounded Benchmark for LLMs Bias Evaluation in Indonesian Languages](https://mlsanity.com/paper/2606.01260/) (2026)

## Abstract (arXiv)

Sailor2 is a family of cutting-edge multilingual language models for South-East Asian (SEA) languages, available in 1B, 8B, and 20B sizes to suit diverse applications. Building on Qwen2.5, Sailor2 undergoes continuous pre-training on 500B tokens (400B SEA-specific and 100B replay tokens) to support 13 SEA languages while retaining proficiency in Chinese and English. Sailor2-20B model achieves a 50-50 win rate against GPT-4o across SEA languages. We also deliver a comprehensive cookbook on how to develop the multilingual model in an efficient manner, including five key aspects: data curation, pre-training, post-training, model customization and evaluation. We hope that Sailor2 model (Apache 2.0 license) will drive language development in the SEA region, and Sailor2 cookbook will inspire researchers to build more inclusive LLMs for other under-served languages.