mlsanityData to 14 Oct 2026

TinyLlama: An Open-Source Small Language Model

arXiv 2401.02385 ↗First posted 4 Jan 2024cs.CL cs.AIcited by 618

Not on the map. Why?

Cited by 50 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present TinyLlama, a compact 1.1B language model pretrained on around 1 trillion tokens for approximately 3 epochs. Building on the architecture and tokenizer of Llama 2, TinyLlama leverages various advances contributed by the open-source community (e.g., FlashAttention and Lit-GPT), achieving better computational efficiency. Despite its relatively small size, TinyLlama demonstrates remarkable performance in a series of downstream tasks. It significantly outperforms existing open-source language models with comparable sizes. Our model checkpoints and code are publicly available on GitHub at https://github.com/jzhang38/TinyLlama.

Key facts. Cited by 618 papers in the map (data through 14 Oct 2026). Builds on 34 of its 44 references that are in the map. First posted on arXiv 4 Jan 2024.

Builds on

34 of 44 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
22 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  4. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  13. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  14. 2023StarCoder: may the source be with you!cited by 799
  15. 2016Language Modeling with Gated Convolutional Networkscited by 702
  16. 2023Scaling Data-Constrained Language Modelscited by 403
  17. 2023Llemma: An Open Language Model For Mathematicscited by 388
  18. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  19. 2023CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-Xcited by 300
  20. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  21. 2023Skywork: A More Open Bilingual Foundation Modelcited by 112
  22. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 246; 2025: 204; 2026: 168.

618 papers.

Sort:

By area:

  1. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  2. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  3. 2024Large Language Models: A Surveycited by 525
  4. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  5. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  6. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  7. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  8. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  9. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  10. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  11. 2024Scaling up Masked Diffusion Models on Textcited by 199
  12. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
38 more
  1. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  2. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  3. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Modelscited by 164
  4. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  5. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  6. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
  7. 2024SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Modelscited by 131
  8. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
  9. 2024Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabulariescited by 118
  10. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  11. 2024EuroLLM: Multilingual Language Models for Europecited by 106
  12. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  13. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 103
  14. 2024Revisiting Catastrophic Forgetting in Large Language Model Tuningcited by 99
  15. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  16. 2024Efficient Multimodal Large Language Models: A Surveycited by 96
  17. 2024Large Language Models and Games: A Survey and Roadmapcited by 93
  18. 2024LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuningcited by 89
  19. 2024Loki: Low-rank Keys for Efficient Sparse Attentioncited by 89
  20. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
  21. 2024OneBit: Towards Extremely Low-bit Large Language Modelscited by 79
  22. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 78
  23. 2024Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs Evaluation, Benchmark, and Arenacited by 76
  24. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 75
  25. 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 75
  26. 2024Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Modelscited by 65
  27. 2024Embodied Understanding of Driving Scenarioscited by 63
  28. 2024Layer-Condensed KV Cache for Efficient Inference of Large Language Modelscited by 59
  29. 2024TinyAgent: Function Calling at the Edgecited by 58
  30. 2024MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Expertscited by 56
  31. 2024HOBBIT: A Mixed Precision Expert Offloading System for Fast MoE Inferencecited by 55
  32. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 54
  33. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 53
  34. 2024Language Models Resist Alignment: Evidence From Data Compressioncited by 53
  35. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 51
  36. 2025MegaMath: Pushing the Limits of Open Math Corporacited by 51
  37. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 50
  38. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priorscited by 49

50 most cited of 618 All 618 (JSON)