mlsanityData to 7 Oct 2026

Stable LM 2 1.6B Technical Report

arXiv 2402.17834 ↗First posted 27 Feb 2024cs.CL stat.MLcited by 86

Not on the map. Why?

Abstract (arXiv)

We introduce StableLM 2 1.6B, the first in a new generation of our language model series. In this technical report, we present in detail the data and training procedure leading to the base and instruction-tuned versions of StableLM 2 1.6B. The weights for both models are available via Hugging Face for anyone to download and use. The report contains thorough evaluations of these models, including zero- and few-shot benchmarks, multilingual benchmarks, and the MT benchmark focusing on multi-turn dialogues. At the time of publishing this report, StableLM 2 1.6B was the state-of-the-art open model under 2B parameters by a significant margin. Given its appealing small size, we also provide throughput measurements on a number of edge devices. In addition, we open source several quantized checkpoints and provide their performance metrics compared to the original model.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 36 of its 62 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

36 of 62 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2016Layer Normalizationcited by 5,579
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
24 more
  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  3. 2019Root Mean Square Layer Normalizationcited by 1,425
  4. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  7. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  8. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  9. 2023Textbooks Are All You Needcited by 561
  10. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  11. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
  12. 2018An Empirical Model of Large-Batch Trainingcited by 268
  13. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  14. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  15. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 183
  16. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 182
  17. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  18. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  19. 2022Testing the Ability of Language Models to Interpret Figurative Languagecited by 67
  20. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 65
  21. 2020Temporal Reasoning on Implicit Events from Distant Supervisioncited by 58
  22. 2022"John is 50 years old, can his son be 65?" Evaluating NLP Models' Understanding of Feasibilitycited by 15
  23. 2023Efficient Parallelization Layouts for Large-Scale Distributed Model Trainingcited by 8
  24. 2023Solving and Generating NPR Sunday Puzzles with Large Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 26; 2026: 12.

86 papers.

Sort:

By area:

  1. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  3. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  4. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  5. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  6. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  7. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  8. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  9. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  10. 2024Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inferencecited by 97
  11. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  12. 2024Resolving Discrepancies in Compute-Optimal Scaling of Language Modelscited by 84
38 more
  1. 2024Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledgecited by 49
  2. 2024Dense Connector for MLLMscited by 48
  3. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 45
  4. 2024Binary Classifier Optimization for Large Language Model Alignmentcited by 44
  5. 2024Memory3: Language Modeling with Explicit Memorycited by 39
  6. 2024Densing Law of LLMscited by 39
  7. 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
  8. 2024Training on the Test Task Confounds Evaluation and Emergencecited by 33
  9. 2024African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classificationcited by 31
  10. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  11. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  12. 2025Taming Knowledge Conflicts in Language Modelscited by 27
  13. 2024How to Synthesize Text Data without Model Collapse?cited by 24
  14. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  15. 2024Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMscited by 21
  16. 2024Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to Seecited by 21
  17. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  18. 2024Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devicescited by 17
  19. 2024RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answeringcited by 15
  20. 2024A Survey of Large Language Models for Arabic Language and its Dialectscited by 13
  21. 2025Eve: Efficient Multimodal Vision Language Models with Elastic Visual Expertscited by 11
  22. 2025Distillation Robustifies Unlearningcited by 11
  23. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10
  24. 2024Are Large Language Models Really Bias-Free? Jailbreak Prompts for Assessing Adversarial Robustness to Bias Elicitationcited by 9
  25. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  26. 2024fMRI predictors based on language models of increasing complexity recover brain left lateralizationcited by 8
  27. 2025Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferencescited by 7
  28. 2025C3PO: Critical-Layer, Core-Expert, Collaborative Pathway Optimization for Test-Time Expert Re-Mixingcited by 7
  29. 2024RedStone: Curating General, Code, Math, and QA Data for Large Language Modelscited by 6
  30. 2025From ASR to ASP: Evaluating Prompt Attack Vulnerabilities Against Open-Source LLMscited by 6
  31. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Modelcited by 6
  32. 2024YuLan-Mini: An Open Data-efficient Language Modelcited by 5
  33. 2025EvoMoE: Expert Evolution in Mixture of Experts for Multimodal Large Language Modelscited by 5
  34. 2025SocialNav-MoE: A Mixture-of-Experts Vision Language Model for Socially Compliant Navigation with Reinforcement Fine-Tuningcited by 5
  35. 2024Mixture-of-Instructions: Aligning Large Language Models via Mixture Promptingcited by 4
  36. 2024Text-only Synthesis for Image Captioningcited by 4
  37. 2024TorchOpera: A Compound AI System for LLM Safetycited by 4
  38. 2024BenTo: Benchmark Task Reduction with In-Context Transferabilitycited by 4

50 most cited of 86 All 86 (JSON)