mlsanityData to 14 Oct 2026

OpenELM: An Efficient Language Model Family with Open Training and Inference Framework

arXiv 2404.14619 ↗First posted 22 Apr 2024cs.CL cs.AI cs.LGcited by 75

Not on the map. Why?

Abstract (arXiv)

The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks. To this end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of approximately one billion parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo while requiring 2× fewer pre-training tokens. Diverging from prior practices that only provide model weights and inference code, and pre-train on private datasets, our release includes the complete framework for training and evaluation of the language model on publicly available datasets, including training logs, multiple checkpoints, and pre-training configurations. We also release code to convert models to MLX library for inference and fine-tuning on Apple devices. This comprehensive release aims to empower and strengthen the open research community, paving the way for future open research endeavors. Our source code along with pre-trained model weights and training recipes is available at urlhttps://github.com/apple/corenet. Additionally, model models can be found on HuggingFace at: urlhttps://huggingface.co/apple/OpenELM.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 39 of its 54 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

39 of 54 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2016Layer Normalizationcited by 5,669
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
27 more
  1. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  2. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  6. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  12. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  13. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  14. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  15. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  16. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  17. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  18. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  19. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  20. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  21. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  22. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  23. 2016Using the Output Embedding to Improve Language Modelscited by 272
  24. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  25. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 104
  26. 2024MobiLlama: Towards Accurate and Lightweight Fully Transparent GPTcited by 34
  27. 2022CVNets: High Performance Library for Computer Visioncited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 26; 2025: 37; 2026: 12.

75 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  2. 2024RedPajama: an Open Dataset for Training Large Language Modelscited by 182
  3. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 177
  4. 2024On-Device Language Models: A Comprehensive Reviewcited by 104
  5. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  6. 2025The Curse of Depth in Large Language Modelscited by 62
  7. 2025LLM-JEPA: Large Language Models Meet Joint Embedding Predictive Architecturescited by 38
  8. 2025Towards Reasoning Ability of Small Language Modelscited by 29
  9. 2024OpenECAD: An Efficient Visual Language Model for Editable 3D-CAD Designcited by 26
  10. 2025Holistically Evaluating the Environmental Impact of Creating Language Modelscited by 26
  11. 2025Minions: Cost-efficient Collaboration Between On-device and Cloud Language Modelscited by 20
  12. 2024Multilingual Large Language Models: A Systematic Surveycited by 18
38 more
  1. 2025Large Language Models for Computer-Aided Design: A Surveycited by 18
  2. 2024Beyond Scaling Laws: Understanding Transformer Performance with Associative Memorycited by 12
  3. 2024Octopus v4: Graph of language modelscited by 11
  4. 2024Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignmentcited by 11
  5. 2024KV Prediction for Improved Time to First Tokencited by 11
  6. 2025Crisp: Cognitive Restructuring of Negative Thoughts through Multi-turn Supportive Dialoguescited by 11
  7. 2025Beyond Text Compression: Evaluating Tokenizers Across Scalescited by 11
  8. 2025EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devicescited by 11
  9. 2024VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Modelscited by 10
  10. 2026PhoStream: Benchmarking Real-World Streaming for Omnimodal Assistants in Mobile Scenarioscited by 10
  11. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 8
  12. 2025Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferencescited by 8
  13. 2025AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimizationcited by 8
  14. 2025AlignHuman: Improving Motion and Fidelity via Timestep-Segment Preference Optimization for Audio-Driven Human Animationcited by 8
  15. 2025Quantized-Tinyllava: a new multimodal foundation model enables efficient split learningcited by 8
  16. 2024On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoningcited by 7
  17. 2024Training Language Models on Synthetic Edit Sequences Improves Code Synthesiscited by 7
  18. 2024Toxicity of the Commons: Curating Open-Source Pre-Training Datacited by 7
  19. 2025GneissWeb: Preparing High Quality Data for LLMs at Scalecited by 7
  20. 2026Semantic Tube Prediction: Beating LLM Data Efficiency with JEPAcited by 7
  21. 2025Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transitioncited by 6
  22. 2026From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AIcited by 6
  23. 2024Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysiscited by 5
  24. 2024Weighted Grouped Query Attention in Transformerscited by 5
  25. 2024SQFT: Low-cost Model Adaptation in Low-precision Sparse Foundation Modelscited by 5
  26. 2025Lossless Compression of Large Language Model-Generated Text via Next-Token Predictioncited by 5
  27. 2025KVSwap: Disk-aware KV Cache Offloading for Long-Context On-device Inferencecited by 5
  28. 2024TorchOpera: A Compound AI System for LLM Safetycited by 4
  29. 2024The Fair Language Model Paradoxcited by 4
  30. 2024Compliance Cards: Automated EU AI Act Compliance Analyses amidst a Complex AI Supply Chaincited by 3
  31. 2025FantasyTalking2: Timestep-Layer Adaptive Preference Optimization for Audio-Driven Portrait Animationcited by 3
  32. 2026Strong Teacher Not Needed? On Distillation in LLM Pretrainingcited by 3
  33. 2024ALLoRA: Adaptive Learning Rate Mitigates LoRA Fatal Flawscited by 2
  34. 2025Small Language Models (SLMs) Can Still Pack a Punch: A survey (updated 2026)cited by 2
  35. 2025SmartBench: Is Your LLM Truly a Good Chinese Smartphone Assistant?cited by 2
  36. 2025Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectivescited by 2
  37. 2025LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Modelcited by 2
  38. 2025Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Trainingcited by 2

50 most cited of 75 All 75 (JSON)