mlsanityData to 7 Oct 2026

The Open Source Advantage in Large Language Models (LLMs)

arXiv 2412.12004 ↗First posted 16 Dec 2024cs.CL cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have rapidly advanced natural language processing, driving significant breakthroughs in tasks such as text generation, machine translation, and domain-specific reasoning. The field now faces a critical dilemma in its approach: closed-source models like GPT-4 deliver state-of-the-art performance but restrict reproducibility, accessibility, and external oversight, while open-source frameworks like LLaMA and Mixtral democratize access, foster collaboration, and support diverse applications, achieving competitive results through techniques like instruction tuning and LoRA. Hybrid approaches address challenges like bias mitigation and resource accessibility by combining the scalability of closed-source systems with the transparency and inclusivity of open-source framework. However, in this position paper, we argue that open-source remains the most robust path for advancing LLM research and ethical deployment.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 60 of its 105 references that are in the map. First posted on arXiv 16 Dec 2024.

Builds on

60 of 105 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  9. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  10. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2020Scaling Laws for Neural Language Modelscited by 6,411
38 more
  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2020Knowledge Distillation: A Surveycited by 1,578
  9. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  10. 2019Publicly Available Clinical BERT Embeddingscited by 995
  11. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  12. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  13. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  14. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  15. 2023The Falcon Series of Open Language Modelscited by 526
  16. 2024Large Language Models: A Surveycited by 506
  17. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  18. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 299
  19. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  20. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  21. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  22. 2023The False Promise of Imitating Proprietary LLMscited by 254
  23. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  24. 2023From Sparse to Soft Mixtures of Expertscited by 245
  25. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  26. 2023Efficient Large Language Models: A Surveycited by 183
  27. 2023Auditing large language models: a three-layered approachcited by 135
  28. 2024NVLM: Open Frontier-Class Multimodal LLMscited by 124
  29. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
  30. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  31. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  32. 2024Safeguarding Large Language Models: A Surveycited by 73
  33. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunitiescited by 63
  34. 2024A Comparative Study on Reasoning Patterns of OpenAI's o1 Modelcited by 62
  35. 2023Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generatorscited by 56
  36. 2021SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Expertscited by 48
  37. 2024The Ethics of ChatGPT in Medicine and Healthcare: A Systematic Review on Large Language Models (LLMs)cited by 45
  38. 2024Rethinking Data Selection for Supervised Fine-Tuningcited by 35

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 9.

18 papers.

Sort:

By area:

  1. 2025Large language models for automated scholarly paper review: A surveycited by 59
  2. 2025Code Graph Model (CGM): A Graph-Integrated Large Language Model for Repository-Level Software Engineering Taskscited by 27
  3. 2026Yunjue Agent Tech Report: A Fully Reproducible, Zero-Start In-Situ Self-Evolving Agent System for Open-Ended Taskscited by 4
  4. 2026MeMo: Memory as a Modelcited by 4
  5. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agentscited by 2
  6. 2025An Open-Source Dual-Loss Embedding Model for Semantic Retrieval in Higher Educationcited by 1
  7. 2025The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Modelscited by 1
  8. 2025Large Language Models for IT Automation Tasks: Are We There Yet?cited by 1
  9. 2026Prompt Programming for Cultural Bias and Alignment of Large Language Modelscited by 1
  10. 2025Federated Foundation Language Model Post-Training Should Focus on Open-Source Modelscited by 0
  11. 2025LogiPart: Local Large Language Models for Data Exploration at Scale with Logical Partitioningcited by 0
  12. 2025Populism Meets AI: Advancing Populism Research with LLMscited by 0
6 more
  1. 2025Matching Ranks Over Probability Yields Truly Deep Safety Alignmentcited by 0
  2. 2026Chinese Labor Law Large Language Model Benchmarkcited by 0
  3. 2026Application-Driven Pedagogical Knowledge Optimization of Open-Source LLMs via Reinforcement Learning and Supervised Fine-Tuningcited by 0
  4. 2026Agentic Insight Generation in VSM Simulationscited by 0
  5. 2026Generative Artificial Intelligence (GenAI) to convert images of queuing networks into verifiable simulation models: an open-weight LLM workflow approachcited by 0
  6. 2026Model Retirement Creates Reproducibility Risk in Biomedical AI Publicationscited by 0