mlsanityData to 14 Oct 2026

An Overview on Language Models: Recent Developments and Outlook

arXiv 2303.05759 ↗First posted 10 Mar 2023cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

Language modeling studies the probability distributions over strings of texts. It is one of the most fundamental tasks in natural language processing (NLP). It has been widely used in text generation, speech recognition, machine translation, etc. Conventional language models (CLMs) aim to predict the probability of linguistic sequences in a causal manner, while pre-trained language models (PLMs) cover broader concepts and can be used in both causal sequential modeling and fine-tuning for downstream applications. PLMs have their own training paradigms (usually self-supervised) and serve as foundation models in modern NLP systems. This overview paper provides an introduction to both CLMs and PLMs from five aspects, i.e., linguistic units, architectures, training methods, evaluation methods, and applications. Furthermore, we discuss the relationship between CLMs and PLMs and shed light on the future directions of language modeling in the pre-trained era.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Builds on 136 of its 221 references that are in the map. First posted on arXiv 10 Mar 2023.

Builds on

136 of 221 references.

References · 136

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  9. 2018Deep contextualized word representationscited by 2,290
  10. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  13. 2019Language Models as Knowledge Bases?cited by 1,936
  14. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  15. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  16. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  17. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  18. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  19. 2018Hierarchical Neural Story Generationcited by 1,193
  20. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  21. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  22. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  23. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  24. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 956
  25. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  26. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  27. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  28. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  29. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  30. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  31. 2019Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailmentcited by 671
  32. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  33. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  34. 2019Green AIcited by 627
  35. 2019Defending Against Neural Fake Newscited by 559
  36. 2015Listen, Attend and Spellcited by 543
  37. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  38. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528

50 most cited of 136 All 136 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 9; 2024: 11; 2025: 2.

22 papers.

Sort:

By area:

  1. 2023Can LLM-Generated Misinformation Be Detected?cited by 177
  2. 2023SeaEval for Multilingual Foundation Models: From Cross-Lingual Alignment to Cultural Reasoningcited by 94
  3. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  4. 2024Navigating LLM Ethics: Advancements, Challenges, and Future Directionscited by 28
  5. 2024Optimal Design for Reward Modeling in RLHFcited by 25
  6. 2023GPT Models in Construction Industry: Opportunities, Limitations, and a Use Case Validationcited by 21
  7. 2023Bias and Fairness in Chatbots: An Overviewcited by 20
  8. 2023Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understandingcited by 20
  9. 2024Resilience of Large Language Models for Noisy Instructionscited by 20
  10. 2023An Overview on Generative AI at Scale with Edge-Cloud Computingcited by 9
  11. 2024XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inferencecited by 8
  12. 2024The global landscape of academic guidelines for generative AI and Large Language Modelscited by 4
10 more
  1. 2024Stars, Stripes, and Silicon: Unravelling the ChatGPT's All-American, Monochrome, Cis-centric Biascited by 4
  2. 2023Ensemble linear interpolators: The role of ensemblingcited by 3
  3. 2023Utilizing Semantic Textual Similarity for Clinical Survey Data Feature Selectioncited by 2
  4. 2023Instructive Dialogue Summarization with Query Aggregationscited by 2
  5. 2024Knowledge Sharing in Manufacturing using Large Language Models: User Evaluation and Model Benchmarkingcited by 2
  6. 2025Unsupervised Speech Segmentation: A General Approach Using Speech Language Modelscited by 1
  7. 2024GWPT: A Green Word-Embedding-based POS Taggercited by 0
  8. 2024Exploring prompts to elicit memorization in masked language model-based named entity recognitioncited by 0
  9. 2024Word Embedding Dimension Reduction via Weakly-Supervised Feature Selectioncited by 0
  10. 2025Integration of TinyML and LargeML: A Survey of 6G and Beyondcited by 0