mlsanityData to 7 Oct 2026

SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scaling

arXiv 2312.15166 ↗First posted 23 Dec 2023cs.CL cs.AI cs.LGcited by 172

Not on the map. Why?

Abstract (arXiv)

We introduce SOLAR 10.7B, a large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale LLMs, we present a method for scaling LLMs called depth up-scaling (DUS), which encompasses depthwise scaling and continued pretraining. In contrast to other LLM up-scaling methods that use mixture-of-experts, DUS does not require complex changes to train and inference efficiently. We show experimentally that DUS is simple yet effective in scaling up high-performance LLMs from small ones. Building on the DUS model, we additionally present SOLAR 10.7B-Instruct, a variant fine-tuned for instruction-following capabilities, surpassing Mixtral-8x7B-Instruct. SOLAR 10.7B is publicly available under the Apache 2.0 license, promoting broad access and application in the LLM field.

Key facts. Cited by 172 papers in the map (data through 7 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 23 Dec 2023.

Builds on

30 of 53 references.

References · 30

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2020Scaling Laws for Neural Language Modelscited by 6,411
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
18 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  6. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  8. 2023NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmarkcited by 339
  9. 2021Scaling Laws for Transfercited by 265
  10. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  11. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  12. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  13. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  14. 2023Don't Make Your LLM an Evaluation Benchmark Cheatercited by 194
  15. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 81
  16. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  17. 2023Reusing Pretrained Models by Multi-linear Operators for Efficient Trainingcited by 15
  18. 2023Composable Function-preserving Expansions for Transformer Architecturescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 83; 2025: 56; 2026: 33.

172 papers.

Sort:

By area:

  1. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2024A Survey on Large Language Models for Code Generationcited by 646
  5. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  6. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  7. 2024LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuningcited by 133
  8. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  9. 2024Systematic Biases in LLM Simulations of Debatescited by 98
  10. 2024Measuring Political Bias in Large Language Models: What Is Said and How It Is Saidcited by 89
  11. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  12. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
38 more
  1. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  2. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  3. 2024Hidden Persuaders: LLMs' Political Leaning and Their Influence on Voterscited by 71
  4. 2024Enhancing Large Vision Language Models with Self-Training on Image Comprehensioncited by 70
  5. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  6. 2024MedExQA: Medical Question Answering Benchmark with Multiple Explanationscited by 58
  7. 2024Can Large Language Models be Good Emotional Supporter? Mitigating Preference Bias on Emotional Support Conversationcited by 55
  8. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  9. 2024FinTextQA: A Dataset for Long-form Financial Question Answeringcited by 54
  10. 2025LongCat-Flash Technical Reportcited by 54
  11. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  12. 2024sDPO: Don't Use Your Data All at Oncecited by 46
  13. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 42
  14. 2024Routoo: Learning to Route to Large Language Models Effectivelycited by 34
  15. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  16. 2024Match, Compare, or Select? An Investigation of Large Language Models for Entity Matchingcited by 32
  17. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  18. 2024Serial Position Effects of Large Language Modelscited by 30
  19. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  20. 2024Retrieval-augmented generation in multilingual settingscited by 27
  21. 2024Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmarkcited by 23
  22. 2024"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreakcited by 23
  23. 2024KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinationscited by 22
  24. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  25. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  26. 2025Rejected Dialects: Biases Against African American Language in Reward Modelscited by 20
  27. 2024Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Modelscited by 19
  28. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19
  29. 2025Kanana: Compute-efficient Bilingual Language Modelscited by 19
  30. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  31. 2024HyperCLOVA X Technical Reportcited by 18
  32. 2024On the Inductive Bias of Stacking Towards Improving Reasoningcited by 18
  33. 2024One Prompt To Rule Them All: LLMs for Opinion Summary Evaluationcited by 16
  34. 2024Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&Acited by 16
  35. 2025Improving Your Model Ranking on Chatbot Arena by Vote Riggingcited by 16
  36. 2025Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakerscited by 16
  37. 2024Enhancing Clinical Efficiency through LLM: Discharge Note Generation for Cardiac Patientscited by 14
  38. 2024Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Modelscited by 13

50 most cited of 172 All 172 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Leveraging the true depth of LLMs
  2. 2025Generalizing Scaling Laws for Dense and Sparse Large Language Models
  3. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
  4. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
  5. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies
  6. 2026Efficient Scaling of LLM Training with Flexible Context Parallelism
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
  8. 2025Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs