mlsanityData to 7 Oct 2026

1.5-Pints Technical Report: Pretraining in Days, Not Months -- Your Language Model Thrives on Quality Data

arXiv 2408.03506 ↗First posted 7 Aug 2024cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper presents a compute-efficient approach to pre-training a Language Model-the "1.5-Pints"-in only 9 days, while outperforming state-of-the-art models as an instruction-following assistant.Based on MT-Bench (a benchmark that emulates human judgments), 1.5-Pints outperforms Apple's OpenELM and Microsoft's Phi.This is achieved by a carefully curated pre-training dataset of 57 billion tokens, using a mix of automated workflows and manual human review. The selection of the dataset prioritizes content that is considered expository and "textbook-like" to aid the model in reasoning and logical deduction, culminating in its overall ability as a strong and versatile AI model. In terms of the model architecture, we employed a modified Mistral tokenizer, alongside a Llama-2 architecture for wider compatibility. For training, we adopted the methodologies used by StableLM, TinyLlama, and Huggingface Zephyr. 1.5-Pints demonstrates that by focusing on data quality over quantity in LLM training, we can significantly reduce training time and resources required. We believe this approach will not only make pre-training more accessible but also reduce our carbon footprint. Our findings and resources from this research are open-sourced, aiming to facilitate further advancements in the field. The 1.5-Pints model is available in two versions: 2K and 16K context windows.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 55 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

35 of 55 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  9. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  11. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  12. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
23 more
  1. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  2. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  3. 2023Textbooks Are All You Needcited by 561
  4. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
  5. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  6. 2023GPT4Tools: Teaching Large Language Model to Use Tools via Self-instructioncited by 245
  7. 2023ToolkenGPT: Augmenting Frozen Language Models with Massive Tools via Tool Embeddingscited by 238
  8. 2022Transformer Quality in Linear Timecited by 189
  9. 2023Knowledge Graph Prompting for Multi-Document Question Answeringcited by 150
  10. 2022Dataset Pruning: Reducing Training Data by Examining Generalization Influencecited by 132
  11. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  12. 2020A3: Accelerating Attention Mechanisms in Neural Networks with Approximationcited by 97
  13. 2024The Hidden Attention of Mamba Modelscited by 88
  14. 2023At Which Training Stage Does Code Data Help LLMs Reasoning?cited by 82
  15. 2021Combiner: Full Attention Transformer with Sparse Computation Costcited by 64
  16. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  17. 2022XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsingcited by 32
  18. 2023Making Large Language Models Better Data Creatorscited by 28
  19. 2023Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generationcited by 13
  20. 2023LLM Augmented Hierarchical Agentscited by 9
  21. 2023BTR: Binary Token Representations for Efficient Retrieval Augmented Language Modelscited by 5
  22. 2024One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Modelscited by 4
  23. 2021An Optimal Resource Allocator of Elastic Training for Deep Learning Jobs on Cloudcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 2.

6 papers.

  1. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  2. 2024Tucano: Advancing Neural Text Generation for Portuguesecited by 12
  3. 2025Distillation Robustifies Unlearningcited by 11
  4. 2024LSHBloom: Memory-efficient, Extreme-scale Document Deduplicationcited by 5
  5. 2025Reading the unreadable: Creating a dataset of 19th century English newspapers using image-to-text language modelscited by 3
  6. 2024Scrambled text: training Language Models to correct OCR errors using synthetic datacited by 2