mlsanityData to 7 Oct 2026

Skywork: A More Open Bilingual Foundation Model

arXiv 2310.19341 ↗First posted 30 Oct 2023cs.CL cs.AIcited by 110

Not on the map. Why?

Abstract (arXiv)

In this technical report, we present Skywork-13B, a family of large language models (LLMs) trained on a corpus of over 3.2 trillion tokens drawn from both English and Chinese texts. This bilingual foundation model is the most extensively trained and openly published LLMs of comparable size to date. We introduce a two-stage training methodology using a segmented corpus, targeting general purpose training and then domain-specific enhancement training, respectively. We show that our model not only excels on popular benchmarks, but also achieves state of the art performance in Chinese language modeling on diverse domains. Furthermore, we propose a novel leakage detection method, demonstrating that test data contamination is a pressing issue warranting further investigation by the LLM community. To spur future research, we release Skywork-13B along with checkpoints obtained during intermediate stages of the training process. We are also releasing part of our SkyPile corpus, a collection of over 150 billion tokens of web text, which is the largest high quality open Chinese pre-training corpus to date. We hope Skywork-13B and our open corpus will serve as a valuable open-source resource to democratize access to high-quality LLMs.

Key facts. Cited by 110 papers in the map (data through 7 Oct 2026). Builds on 43 of its 56 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

43 of 56 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
31 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  2. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2018Deep contextualized word representationscited by 2,285
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2020GLU Variants Improve Transformercited by 1,724
  13. 2023PaLM 2 Technical Reportcited by 1,154
  14. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  15. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  16. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  17. 2022Galactica: A Large Language Model for Sciencecited by 736
  18. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  19. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  20. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  21. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  22. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  23. 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
  24. 2023Scaling Data-Constrained Language Modelscited by 391
  25. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  26. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  27. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  28. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  29. 2021Scaling Laws for Transfercited by 265
  30. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 147
  31. 2022Training Trajectories of Language Models Across Scalescited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 72; 2025: 22; 2026: 7.

110 papers.

Sort:

By area:

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  2. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  3. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  4. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  5. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  6. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  7. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  8. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  9. 2024mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrievalcited by 233
  10. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  11. 2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 127
  12. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
38 more
  1. 2024AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Promptingcited by 113
  2. 2024Benchmarking Benchmark Leakage in Large Language Modelscited by 102
  3. 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origincited by 92
  4. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86
  5. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
  6. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluationcited by 73
  7. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  8. 2024Skywork-MoE: A Deep Dive into Training Techniques for Mixture-of-Experts Language Modelscited by 57
  9. 2025PHYBench: Holistic Evaluation of Physical Perception and Reasoning in Large Language Modelscited by 57
  10. 2024Compression Represents Intelligence Linearlycited by 56
  11. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  12. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 49
  13. 2024Scaling Laws for Predicting Downstream Performance in LLMscited by 48
  14. 2023Fake Alignment: Are LLMs Really Aligned Well?cited by 39
  15. 2024Densing Law of LLMscited by 39
  16. 2024KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Modelscited by 36
  17. 2025Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Datacited by 35
  18. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
  19. 2024Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluationcited by 33
  20. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densitiescited by 32
  21. 2024History, Development, and Principles of Large Language Models-An Introductory Surveycited by 30
  22. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  23. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encodingcited by 28
  24. 2024Conifer: Improving Complex Constrained Instruction-Following Ability of Large Language Modelscited by 26
  25. 2024Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Modelcited by 25
  26. 2024Is Your LLM Outdated? A Deep Look at Temporal Generalizationcited by 25
  27. 2024Value Residual Learningcited by 25
  28. 2024Skywork-Math: Data Scaling Laws for Mathematical Reasoning in Large Language Models -- The Story Goes Oncited by 24
  29. 2024Training on the Benchmark Is Not All You Needcited by 24
  30. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  31. 2024Astraios: Parameter-Efficient Instruction Tuning Code Large Language Modelscited by 23
  32. 2024Unveiling Linguistic Regions in Large Language Modelscited by 22
  33. 2025AURORA:Automated Training Framework of Universal Process Reward Models via Ensemble Prompting and Reverse Verificationcited by 22
  34. 2023RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Modelscited by 21
  35. 2024Sailor: Open Language Models for South-East Asiacited by 19
  36. 2024FormalAlign: Automated Alignment Evaluation for Autoformalizationcited by 19
  37. 2025Are Large Vision Language Models Good Game Players?cited by 19
  38. 2024From GPT-4 to Gemini and Beyond: Assessing the Landscape of MLLMs on Generalizability, Trustworthiness and Causality through Four Modalitiescited by 18

50 most cited of 110 All 110 (JSON)