mlsanityData to 7 Oct 2026

XGen-7B Technical Report

arXiv 2309.03450 ↗First posted 7 Sep 2023cs.CL cs.AI cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have become ubiquitous across various domains, transforming the way we interact with information and conduct research. However, most high-performing LLMs remain confined behind proprietary walls, hindering scientific progress. Most open-source LLMs, on the other hand, are limited in their ability to support longer sequence lengths, which is a key requirement for many tasks that require inference over an input context. To address this, we have trained XGen, a series of 7B parameter models on up to 8K sequence length for up to 1.5T tokens. We have also finetuned the XGen models on public-domain instructional data, creating their instruction-tuned counterparts (XGen-Inst). We open-source our models for both research advancements and commercial applications. Our evaluation on standard benchmarks shows that XGen models achieve comparable or better results when compared with state-of-the-art open-source LLMs. Our targeted evaluation on long sequence modeling tasks shows the benefits of our 8K-sequence models over 2K-sequence open-source LLMs.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

31 of 44 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
19 more
  1. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  6. 2019Root Mean Square Layer Normalizationcited by 1,425
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2022CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesiscited by 876
  10. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  11. 2023StarCoder: may the source be with you!cited by 790
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  13. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  14. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  15. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  16. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  17. 2021SummScreen: A Dataset for Abstractive Screenplay Summarizationcited by 80
  18. 2023A Theory on Adam Instability in Large-Scale Machine Learningcited by 54
  19. 2020BARD: A structured technique for group elicitation of Bayesian networks to support analytic reasoningcited by 13

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 5; 2025: 2.

11 papers.

  1. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 443
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  3. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  4. 2024LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inferencecited by 70
  5. 2025WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMscited by 25
  6. 2023S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Modelscited by 18
  7. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?cited by 15
  8. 2024Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuningcited by 14
  9. 2025AdmTree: Compressing Lengthy Context with Adaptive Semantic Treescited by 7
  10. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Textcited by 5
  11. 2024ViUniT: Visual Unit Tests for More Robust Visual Programmingcited by 3