mlsanityData to 14 Oct 2026

xGen-small Technical Report

arXiv 2505.06496 ↗First posted 10 May 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

We introduce xGen-small, a family of 4B and 9B Transformer decoder models optimized for long-context applications. Our vertically integrated pipeline unites domain-balanced, frequency-aware data curation; multi-stage pre-training with quality annealing and length extension to 128k tokens; and targeted post-training via supervised fine-tuning, preference learning, and online reinforcement learning. xGen-small delivers strong performance across various tasks, especially in math and coding domains, while excelling at long context benchmarks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 30 of its 33 references that are in the map. First posted on arXiv 10 May 2025.

Builds on

30 of 33 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2024Qwen2.5 Technical Reportcited by 4,942
18 more
  1. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  3. 2021Program Synthesis with Large Language Modelscited by 3,637
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  8. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  12. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  13. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  14. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  15. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  16. 2022Scaling Laws and Interpretability of Learning from Repeated Datacited by 152
  17. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  18. 2024Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Schedulercited by 28

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Mid-Training of Large Language Models: A Surveycited by 21

Similar, not cited

Close in wording, no citation either way

  1. 2023Accelerating Large Language Model Decoding with Speculative Sampling
  2. 2026Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling
  3. 2022Efficiently Scaling Transformer Inference
  4. 2024Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization
  5. 2025Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models
  6. 2026The Expressive Power of Low Precision Softmax Transformers with (Summarized) Chain-of-Thought
  7. 2024Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding
  8. 2025Transformers in Pseudo-Random Number Generation: A Dual Perspective on Theory and Practice