mlsanityData to 14 Oct 2026

Xmodel-LM Technical Report

arXiv 2406.02856 ↗First posted 5 Jun 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We introduce Xmodel-LM, a compact and efficient 1.1B language model pre-trained on around 2 trillion tokens. Trained on our self-built dataset (Xdata), which balances Chinese and English corpora based on downstream task optimization, Xmodel-LM exhibits remarkable performance despite its smaller size. It notably surpasses existing open-source language models of similar scale. Our model checkpoints and code are publicly accessible on GitHub at https://github.com/XiaoduoAILab/XmodelLM.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 31 references that are in the map. First posted on arXiv 5 Jun 2024.

Builds on

22 of 31 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  8. 2023Qwen Technical Reportcited by 3,344
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
10 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  9. 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
  10. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333

Built on by

Not cited yet.