mlsanityData to 14 Oct 2026

Fox-1: Open Small Language Model for Cloud and Edge

arXiv 2411.05281 ↗First posted 8 Nov 2024cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

We present Fox-1, a series of small language models (SLMs) consisting of Fox-1-1.6B and Fox-1-1.6B-Instruct-v0.1. These models are pre-trained on 3 trillion tokens of web-scraped document data and fine-tuned with 5 billion tokens of instruction-following and multi-turn conversation data. Aiming to improve the pre-training efficiency, Fox-1-1.6B model introduces a novel 3-stage data curriculum across all the training data with 2K-8K sequence length. In architecture design, Fox-1 features a deeper layer structure, an expanded vocabulary, and utilizes Grouped Query Attention (GQA), offering a performant and efficient architecture compared to other SLMs. Fox-1 achieves better or on-par performance in various benchmarks compared to StableLM-2-1.6B, Gemma-2B, Qwen1.5-1.8B, and OpenELM1.1B, with competitive inference speed and throughput. The model weights have been released under the Apache 2.0 license, where we aim to promote the democratization of LLMs and make them fully accessible to the whole open-source community.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 8 Nov 2024.

Builds on

49 of 58 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  12. 2020Scaling Laws for Neural Language Modelscited by 6,586
37 more
  1. 2016Layer Normalizationcited by 5,669
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  13. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  14. 2024Mixtral of Expertscited by 1,823
  15. 2020GLU Variants Improve Transformercited by 1,804
  16. 2019Root Mean Square Layer Normalizationcited by 1,480
  17. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  18. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  19. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  20. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  21. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  22. 2016Wide & Deep Learning for Recommender Systemscited by 747
  23. 2019Compressive Transformers for Long-Range Sequence Modellingcited by 726
  24. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  25. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  26. 2023Textbooks Are All You Needcited by 569
  27. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  28. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  29. 2022The Stack: 3 TB of permissively licensed source codecited by 359
  30. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  31. 2023OpenWebMath: An Open Dataset of High-Quality Mathematical Web Textcited by 155
  32. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  33. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  34. 2023Branch-Solve-Merge Improves Large Language Model Evaluation and Generationcited by 106
  35. 2024Stable LM 2 1.6B Technical Reportcited by 90
  36. 2023System 2 Attention (is something you might need too)cited by 89
  37. 2023LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Modelscited by 60

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Unbiased Gradient Low-Rank Projectioncited by 7
  2. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  3. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0