# History, Development, and Principles of Large Language Models-An Introductory Survey

arXiv: https://arxiv.org/abs/2402.06853 · HTML: https://mlsanity.com/paper/2402.06853/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 97 of its 177 references that are in the map.
- First posted on arXiv 10 Feb 2024.

## Builds on (most cited first, 30 of 97)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)

## Built on by (most cited first, 30 of 30)

- [Fairness in Large Language Models: A Taxonomic Survey](https://mlsanity.com/paper/2404.01349/) (2024)
- [A Survey on Human Preference Learning for Large Language Models](https://mlsanity.com/paper/2406.11191/) (2024)
- [Multi-Head RAG: Solving Multi-Aspect Problems with LLMs](https://mlsanity.com/paper/2406.05085/) (2024)
- [Fairness Definitions in Language Models Explained](https://mlsanity.com/paper/2407.18454/) (2024)
- [Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2503.16585/) (2025)
- [Digital Forensics in the Age of Large Language Models](https://mlsanity.com/paper/2504.02963/) (2025)
- [Fairness in Large Language Models in Three Hours](https://mlsanity.com/paper/2408.00992/) (2024)
- [Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications](https://mlsanity.com/paper/2510.21887/) (2025)
- [Towards Harnessing the Collaborative Power of Large and Small Models for Domain Tasks](https://mlsanity.com/paper/2504.17421/) (2025)
- [SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures](https://mlsanity.com/paper/2510.08942/) (2025)
- [AMCR: A Framework for Assessing and Mitigating Copyright Risks in Generative Models](https://mlsanity.com/paper/2509.00641/) (2025)
- [Towards Transparent AI: A Survey on Explainable Language Models](https://mlsanity.com/paper/2509.21631/) (2025)
- [When Do Tools and Planning Help Large Language Models Think? A Cost- and Latency-Aware Benchmark](https://mlsanity.com/paper/2601.02663/) (2026)
- [Multi-Agent Home Energy Management Assistant](https://mlsanity.com/paper/2602.15219/) (2026)
- [SRLCG: Self-Rectified Large-Scale Code Generation with Multidimensional Chain-of-Thought and Dynamic Backtracking](https://mlsanity.com/paper/2504.00532/) (2025)
- [When AI Fails, What Works? A Data-Driven Taxonomy of Real-World AI Risk Mitigation Strategies](https://mlsanity.com/paper/2603.04259/) (2026)
- [Uncertain Boundaries: Multidisciplinary Approaches to Copyright Issues in Generative AI](https://mlsanity.com/paper/2404.08221/) (2024)
- [Educational-Psychological Dialogue Robot Based on Multi-Agent Collaboration](https://mlsanity.com/paper/2412.03847/) (2024)
- [Recent Advances and Future Directions in Literature-Based Discovery](https://mlsanity.com/paper/2506.12385/) (2025)
- [How LLMs are Shaping the Future of Virtual Reality](https://mlsanity.com/paper/2508.00737/) (2025)
- [Vision-Enabled LLMs in Historical Lexicography: Digitising and Enriching Estonian-German Dictionaries from the 17th and 18th Centuries](https://mlsanity.com/paper/2510.07931/) (2025)
- [Simulated Self-Assessment in Large Language Models: A Psychometric Approach to AI Self-Efficacy](https://mlsanity.com/paper/2511.19872/) (2025)
- [CABENCH: Benchmarking Composable AI for Solving Complex Tasks through Composing Ready-to-Use Models](https://mlsanity.com/paper/2508.02427/) (2025)
- [Beyond Technocratic XAI: The Who, What & How in Explanation Design](https://mlsanity.com/paper/2508.09231/) (2025)
- [Language-Guided Tuning: Configuration Optimization for Automated ML Research](https://mlsanity.com/paper/2508.15757/) (2025)
- [Evaluating LLM-Based Process Explanations under Progressive Behavioral-Input Reduction](https://mlsanity.com/paper/2510.09732/) (2025)
- [Reasoning Capabilities of Large Language Models. Lessons Learned from General Game Playing](https://mlsanity.com/paper/2602.19160/) (2026)
- [DART: A DAG-Based Reputation and Incentive Framework via Blockchain-Enabled Governance for Trustworthy LLM Multi-Agent Collaboration](https://mlsanity.com/paper/2609.05529/) (2026)
- [The Life of a Token: from Words to Bits on the Wire](https://mlsanity.com/paper/2609.19924/) (2026)
- [FairLMs: A Turnkey Library for Fairness in Language Models](https://mlsanity.com/paper/2609.21296/) (2026)

## Abstract (arXiv)

Language models serve as a cornerstone in natural language processing (NLP), utilizing mathematical methods to generalize language laws and knowledge for prediction and generation. Over extensive research spanning decades, language modeling has progressed from initial statistical language models (SLMs) to the contemporary landscape of large language models (LLMs). Notably, the swift evolution of LLMs has reached the ability to process, understand, and generate human-level text. Nevertheless, despite the significant advantages that LLMs offer in improving both work and personal lives, the limited understanding among general practitioners about the background and principles of these models hampers their full potential. Notably, most LLM reviews focus on specific aspects and utilize specialized language, posing a challenge for practitioners lacking relevant background knowledge. In light of this, this survey aims to present a comprehensible overview of LLMs to assist a broader audience. It strives to facilitate a comprehensive understanding by exploring the historical background of language models and tracing their evolution over time. The survey further investigates the factors influencing the development of LLMs, emphasizing key contributions. Additionally, it concentrates on elucidating the underlying principles of LLMs, equipping audiences with essential theoretical knowledge. The survey also highlights the limitations of existing work and points out promising future directions.