mlsanityData to 7 Oct 2026

Decoding Large-Language Models: A Systematic Overview of Socio-Technical Impacts, Constraints, and Emerging Questions

arXiv 2409.16974 ↗First posted 25 Sep 2024cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

There have been rapid advancements in the capabilities of large language models (LLMs) in recent years, greatly revolutionizing the field of natural language processing (NLP) and artificial intelligence (AI) to understand and interact with human language. Therefore, in this work, we conduct a systematic investigation of the literature to identify the prominent themes and directions of LLM developments, impacts, and limitations. Our findings illustrate the aims, methodologies, limitations, and future directions of LLM research. It includes responsible development considerations, algorithmic improvements, ethical challenges, and societal implications of LLM development. Overall, this paper provides a rigorous and comprehensive overview of current research in LLM and identifies potential directions for future development. The article highlights the application areas that could have a positive impact on society along with the ethical considerations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 49 of its 65 references that are in the map. First posted on arXiv 25 Sep 2024.

Builds on

49 of 65 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
37 more
  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2019Language Models as Knowledge Bases?cited by 1,922
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  13. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  14. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  15. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  16. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  17. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  18. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  19. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  20. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  21. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  22. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  23. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  24. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  25. 2016Language Modeling with Gated Convolutional Networkscited by 688
  26. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  27. 2023A Watermark for Large Language Modelscited by 667
  28. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  29. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 525
  30. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  31. 2019Release Strategies and the Social Impacts of Language Modelscited by 505
  32. 2023VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Taskscited by 499
  33. 2023Large Language Models Are Zero-Shot Time Series Forecasterscited by 491
  34. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  35. 2016Using the Output Embedding to Improve Language Modelscited by 266
  36. 2021How Many Data Points is a Prompt Worth?cited by 197
  37. 2016Exploring the Limits of Language Modelingcited by 172

Built on by

Not cited yet.