# A Bibliometric Review of Large Language Models Research from 2017 to 2023

arXiv: https://arxiv.org/abs/2304.02020 · HTML: https://mlsanity.com/paper/2304.02020/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 94 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [BERTweet: A pre-trained language model for English Tweets](https://mlsanity.com/paper/2005.10200/) (2020)
- [GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models](https://mlsanity.com/paper/2303.10130/) (2023)
- [FlauBERT: Unsupervised Language Model Pre-training for French](https://mlsanity.com/paper/1912.05372/) (2019)
- [Pre-training of Graph Augmented Transformers for Medication Recommendation](https://mlsanity.com/paper/1906.00346/) (2019)
- [What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers](https://mlsanity.com/paper/2109.04650/) (2021)

## Built on by (most cited first, 30 of 57)

- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes](https://mlsanity.com/paper/2312.14890/) (2023)
- [Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision](https://mlsanity.com/paper/2310.18689/) (2023)
- [A Comprehensive Survey on Segment Anything Model for Vision and Beyond](https://mlsanity.com/paper/2305.08196/) (2023)
- ["HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media](https://mlsanity.com/paper/2304.10619/) (2023)
- [chatClimate: Grounding Conversational AI in Climate Science](https://mlsanity.com/paper/2304.05510/) (2023)
- [BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis](https://mlsanity.com/paper/2404.15532/) (2024)
- [When Your Reviewer is an LLM: Biases, Divergence, and Prompt Injection Risks in Peer Review](https://mlsanity.com/paper/2509.09912/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Can LLMs like GPT-4 outperform traditional AI tools in dementia diagnosis? Maybe, but not today](https://mlsanity.com/paper/2306.01499/) (2023)
- [Simulated patient systems powered by large language model-based AI agents offer potential for transforming medical education](https://mlsanity.com/paper/2409.18924/) (2024)
- [Explore the Potential of LLMs in Misinformation Detection: An Empirical Study](https://mlsanity.com/paper/2311.12699/) (2023)
- [WorldMedQA-V: a multilingual, multimodal medical examination dataset for multimodal language models evaluation](https://mlsanity.com/paper/2410.12722/) (2024)
- [CORE-GPT: Combining Open Access research and large language models for credible, trustworthy question answering](https://mlsanity.com/paper/2307.04683/) (2023)
- [Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis](https://mlsanity.com/paper/2403.16303/) (2024)
- [Evaluation of AI Chatbots for Patient-Specific EHR Questions](https://mlsanity.com/paper/2306.02549/) (2023)
- [An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasets](https://mlsanity.com/paper/2403.15230/) (2024)
- [Financial Sentiment Analysis on News and Reports Using Large Language Models and FinBERT](https://mlsanity.com/paper/2410.01987/) (2024)
- [ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmaps](https://mlsanity.com/paper/2305.06472/) (2023)
- [Evaluating Cultural Awareness of LLMs for Yoruba, Malayalam, and English](https://mlsanity.com/paper/2410.01811/) (2024)
- [Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiency](https://mlsanity.com/paper/2402.19366/) (2024)
- [ChatGPT v Bard v Bing v Claude 2 v Aria v human-expert. How good are AI chatbots at scientific writing?](https://mlsanity.com/paper/2309.08636/) (2023)
- [QuIM-RAG: Advancing Retrieval-Augmented Generation with Inverted Question Matching for Enhanced QA Performance](https://mlsanity.com/paper/2501.02702/) (2025)
- [NPHardEval4V: Dynamic Evaluation of Large Vision-Language Models with Effects of Vision](https://mlsanity.com/paper/2403.01777/) (2024)
- [Segment Anything for Videos: A Systematic Survey](https://mlsanity.com/paper/2408.08315/) (2024)
- [Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models](https://mlsanity.com/paper/2508.17184/) (2025)
- [Utilizing ChatGPT to Enhance Clinical Trial Enrollment](https://mlsanity.com/paper/2306.02077/) (2023)
- [KAHANI: Culturally-Nuanced Visual Storytelling Tool for Non-Western Cultures](https://mlsanity.com/paper/2410.19419/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are a class of language models that have demonstrated outstanding performance across a range of natural language processing (NLP) tasks and have become a highly sought-after research area, because of their ability to generate human-like language and their potential to revolutionize science and technology. In this study, we conduct bibliometric and discourse analyses of scholarly literature on LLMs. Synthesizing over 5,000 publications, this paper serves as a roadmap for researchers, practitioners, and policymakers to navigate the current landscape of LLMs research. We present the research trends from 2017 to early 2023, identifying patterns in research paradigms and collaborations. We start with analyzing the core algorithm developments and NLP tasks that are fundamental in LLMs research. We then investigate the applications of LLMs in various fields and domains including medicine, engineering, social science, and humanities. Our review also reveals the dynamic, fast-paced evolution of LLMs research. Overall, this paper offers valuable insights into the current state, impact, and potential of LLMs research and its applications.