mlsanityData to 14 Oct 2026

ChuXin: 1.6B Technical Report

arXiv 2405.04828 ↗First posted 8 May 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this report, we present ChuXin, an entirely open-source language model with a size of 1.6 billion parameters. Unlike the majority of works that only open-sourced the model weights and architecture, we have made everything needed to train a model available, including the training data, the training process, and the evaluation code. Our goal is to empower and strengthen the open research community, fostering transparency and enabling a new wave of innovation in the field of language modeling. Furthermore, we extend the context length to 1M tokens through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. The weights for both models are available at Hugging Face to download and use.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 47 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

41 of 47 references.

References · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2023Qwen Technical Reportcited by 3,344
  11. 2023Mistral 7Bcited by 3,164
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
29 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  2. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  3. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  4. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  5. 2020GLU Variants Improve Transformercited by 1,804
  6. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  7. 2019Root Mean Square Layer Normalizationcited by 1,480
  8. 2021Scaling Vision Transformerscited by 999
  9. 2023StarCoder: may the source be with you!cited by 799
  10. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  11. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  12. 2024Yi: Open Foundation Models by 01.AIcited by 719
  13. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 708
  14. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  15. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  16. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  17. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  18. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  19. 2023Scaling Data-Constrained Language Modelscited by 403
  20. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  21. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  22. 2020A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languagescited by 131
  23. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  24. 2024Stable LM 2 1.6B Technical Reportcited by 90
  25. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  26. 2023WanJuan: A Comprehensive Multimodal Dataset for Advancing English and Chinese Large Modelscited by 50
  27. 2024Chinese Tiny LLM: Pretraining a Chinese-Centric Large Language Modelcited by 25
  28. 2024H2O-Danube-1.8B Technical Reportcited by 9
  29. 2022reStructured Pre-trainingcited by 8

Built on by

Not cited yet.