mlsanityData to 7 Oct 2026

Orion-14B: Open-source Multilingual Large Language Models

arXiv 2401.12246 ↗First posted 20 Jan 2024cs.CL cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts in English, Chinese, Japanese, Korean, and other languages. Additionally, we fine-tuned a series of models tailored for conversational applications and other specific use cases. Our evaluation results demonstrate that Orion-14B achieves state-of-the-art performance across a broad spectrum of tasks. We make the Orion-14B model family and its associated code publicly accessible https://github.com/OrionStarAI/Orion, aiming to inspire future research and practical applications in the field.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 58 references that are in the map. First posted on arXiv 20 Jan 2024.

Builds on

36 of 58 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2020Scaling Laws for Neural Language Modelscited by 6,411
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
24 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  4. 2018Deep contextualized word representationscited by 2,285
  5. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  13. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  14. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  15. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  16. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 436
  17. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  18. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  19. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  20. 2023Skill-it! A Data-Driven Skills Framework for Understanding and Training Language Modelscited by 120
  21. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  22. 2023Language acquisition: do children and language models follow similar learning stages?cited by 40
  23. 2023A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Modelscited by 22
  24. 2023DotHash: Estimating Set Similarity Metrics for Link Prediction and Document Deduplicationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024LongAlign: A Recipe for Long Context Alignment of Large Language Modelscited by 102
  2. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  3. 2025Preference Learning Unlocks LLMs' Psycho-Counseling Skillscited by 9
  4. 2024Leveraging Large Language Models for Enhanced NLP Task Performance through Knowledge Distillation and Optimized Training Strategiescited by 6
  5. 2026FINEST: Improving LLM Responses to Sensitive Topics Through Fine-Grained Evaluationcited by 0