mlsanityData to 7 Oct 2026

Understanding LLMs: A Comprehensive Overview from Training to Inference

arXiv 2401.02038 ↗First posted 4 Jan 2024cs.CLcited by 58

Not on the map. Why?

Abstract (arXiv)

The introduction of ChatGPT has led to a significant increase in the utilization of Large Language Models (LLMs) for addressing downstream tasks. There's an increasing focus on cost-efficient training and deployment within this context. Low-cost training and deployment of LLMs represent the future development trend. This paper reviews the evolution of large language model training techniques and inference deployment technologies aligned with this emerging trend. The discussion on training includes various aspects, including data preprocessing, training architecture, pre-training tasks, parallel training, and relevant content related to model fine-tuning. On the inference side, the paper covers topics such as model compression, parallel computation, memory scheduling, and structural optimization. It also explores LLMs' utilization and provides insights into their future development.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Builds on 150 of its 203 references that are in the map. First posted on arXiv 4 Jan 2024.

Builds on

150 of 203 references.

References · 150

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2017Proximal Policy Optimization Algorithmscited by 14,441
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  12. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
38 more
  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  11. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  12. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  13. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  14. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  15. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  16. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  17. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  18. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  19. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  20. 2021Program Synthesis with Large Language Modelscited by 3,509
  21. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  22. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  23. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  24. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  25. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  26. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  27. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  28. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  29. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  30. 2023A Survey of Large Language Modelscited by 2,537
  31. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  32. 2018Deep contextualized word representationscited by 2,285
  33. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  34. 2020Extracting Training Data from Large Language Modelscited by 2,144
  35. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  36. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  37. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  38. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968

50 most cited of 150 All 150 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 21; 2026: 7.

58 papers.

Sort:

By area:

  1. 2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
  2. 2024Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQLcited by 118
  3. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  4. 2024Revolutionizing Finance with LLMs: An Overview of Applications and Insightscited by 88
  5. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  6. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  7. 2024Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Researchcited by 26
  8. 2024Large Language Models for Manufacturingcited by 24
  9. 2024Unified Parameter-Efficient Unlearning for LLMscited by 24
  10. 2024Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchangecited by 22
  11. 2025Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUscited by 21
  12. 2024AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuningcited by 16
38 more
  1. 2024ESPACE: Dimensionality Reduction of Activations for Model Compressioncited by 15
  2. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  3. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13
  4. 2024OneEdit: A Neural-Symbolic Collaboratively Knowledge Editing Systemcited by 12
  5. 2024Transcending Language Boundaries: Harnessing LLMs for Low-Resource Language Translationcited by 12
  6. 2024GP-GPT: Large Language Model for Gene-Phenotype Mappingcited by 10
  7. 2024FoldGPT: Simple and Effective Large Language Model Compression Schemecited by 9
  8. 2024Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inferencecited by 9
  9. 2024In2Core: Leveraging Influence Functions for Coreset Selection in Instruction Finetuning of Large Language Modelscited by 9
  10. 2025Procedural Memory Is Not All You Need: Bridging Cognitive Gaps in LLM-Based Agentscited by 9
  11. 2024FLEXTAF: Enhancing Table Reasoning with Flexible Tabular Formatscited by 8
  12. 2024What Do Speech Foundation Models Not Learn About Speech?cited by 8
  13. 2024A Moral Imperative: The Need for Continual Superalignment of Large Language Modelscited by 5
  14. 2024Foundation Models for Low-Resource Language Education (Vision Paper)cited by 5
  15. 2025AI Governance through Marketscited by 5
  16. 2025Brain-Inspired Exploration of Functional Networks and Key Neurons in Large Language Modelscited by 4
  17. 2025Pruning Large Language Models by Identifying and Preserving Functional Networkscited by 4
  18. 2024Revolutionizing Bridge Operation and Maintenance with LLM-based Agents: An Overview of Applications and Insightscited by 3
  19. 2024Recognizing Emotion Regulation Strategies from Human Behavior with Large Language Modelscited by 3
  20. 2025Large Language Models for Bioinformaticscited by 3
  21. 2025LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAMcited by 3
  22. 2024Human-AI Collaborative Taxonomy Construction: A Case Study in Profession-Specific Writing Assistantscited by 2
  23. 2024The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcarecited by 2
  24. 2025Harnessing the Potential of Large Language Models in Modern Marketing Management: Applications, Future Directions, and Strategic Recommendationscited by 2
  25. 2025Experiments with Large Language Models on Retrieval-Augmented Generation for Closed-Source Simulation Softwarecited by 2
  26. 2025Helping Large Language Models Protect Themselves: An Enhanced Filtering and Summarization Systemcited by 2
  27. 2025Predicate Renaming via Large Language Modelscited by 2
  28. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestrationcited by 2
  29. 2024Technical Language Processing for Telecommunications Specificationscited by 1
  30. 2025Reasoning about Affordances: Causal and Compositional Reasoning in LLMscited by 1
  31. 2025REMoH: A Reflective Evolution of Multi-objective Heuristics approach via Large Language Modelscited by 1
  32. 2025LexiMark: Robust Watermarking via Lexical Substitutions to Enhance Membership Verification of an LLM's Textual Training Datacited by 1
  33. 2025A Survey: Towards Privacy and Security in Mobile Large Language Modelscited by 1
  34. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  35. 2024ERASMO: Leveraging Large Language Models for Enhanced Clustering Segmentationcited by 0
  36. 2024An Approach for Auto Generation of Labeling Functions for Software Engineering Chatbotscited by 0
  37. 2024Software Performance Engineering for Foundation Model-Powered Softwarecited by 0
  38. 2025A Deep User Interface for Exploring LLaMacited by 0

50 most cited of 58 All 58 (JSON)