mlsanityData to 14 Oct 2026

MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT

arXiv 2402.16840 ↗First posted 26 Feb 2024cs.CLcited by 34

Not on the map. Why?

Abstract (arXiv)

"Bigger the better" has been the predominant trend in recent Large Language Models (LLMs) development. However, LLMs do not suit well for scenarios that require on-device processing, energy efficiency, low memory footprint, and response efficiency. These requisites are crucial for privacy, security, and sustainable deployment. This paper explores the "less is more" paradigm by addressing the challenge of designing accurate yet efficient Small Language Models (SLMs) for resource constrained devices. Our primary contribution is the introduction of an accurate and fully transparent open-source 0.5 billion (0.5B) parameter SLM, named MobiLlama, catering to the specific needs of resource-constrained computing with an emphasis on enhanced performance with reduced resource demands. MobiLlama is a SLM design that initiates from a larger model and applies a careful parameter sharing scheme to reduce both the pre-training and the deployment cost. Our work strives to not only bridge the gap in open-source SLMs but also ensures full transparency, where complete training data pipeline, training code, model weights, and over 300 checkpoints along with evaluation codes is available at : https://github.com/mbzuai-oryx/MobiLlama.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

29 of 45 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2016Pointer Sentinel Mixture Modelscited by 3,096
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  11. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
17 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2019Towards VQA Models That Can Readcited by 1,718
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  9. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  11. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  12. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  13. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  14. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  15. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  16. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  17. 2021Leveraging redundancy in attention with Reuse Transformerscited by 33

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 14; 2026: 6.

34 papers.

Sort:

By area:

  1. 2024OpenELM: An Efficient Language Model Family with Open Training and Inference Frameworkcited by 75
  2. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  3. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 36
  4. 2024A Survey of Small Language Modelscited by 31
  5. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
  6. 2025HumaniBench: A Human-Centric Framework for Large Multimodal Models Evaluationcited by 19
  7. 2024InkubaLM: A small language model for low-resource African languagescited by 18
  8. 2024TroL: Traversal of Layers for Large Language and Vision Modelscited by 14
  9. 2024VURF: A General-purpose Reasoning and Self-refinement Framework for Video Understandingcited by 13
  10. 2024Super Tiny Language Modelscited by 7
  11. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  12. 2025Thinking Beyond Tokens: From Brain-Inspired Intelligence to Cognitive Foundations for Artificial General Intelligence and its Societal Impactcited by 6
22 more
  1. 2025GAEA: A Geolocation Aware Conversational Assistantcited by 5
  2. 2025A Culturally-diverse Multilingual Multimodal Video Benchmark & Modelcited by 5
  3. 2024Achieving Sparse Activation in Small Language Modelscited by 4
  4. 2024Lillama: Large Language Models Compression via Low-Rank Feature Distillationcited by 4
  5. 2025ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inferencecited by 4
  6. 2024EfficientMorph: Parameter-Efficient Transformer-Based Architecture for 3D Image Registrationcited by 3
  7. 2025CoSMoEs: Compact Sparse Mixture of Expertscited by 3
  8. 2025Mitigating Attention Localization in Small Scale: Self-Attention Refinement via One-step Belief Propagationcited by 3
  9. 2025Beyond the Tip of Efficiency: Uncovering the Submerged Threats of Jailbreak Attacks in Small Language Modelscited by 2
  10. 2025Kolmogorov-Arnold Attention: Is Learnable Attention Better For Vision Transformers?cited by 2
  11. 2025STROKEVISION-BENCH: A Multimodal Video And 2D Pose Benchmark For Tracking Stroke Recoverycited by 2
  12. 2026Learning to Share: Selective Memory for Efficient Parallel Agentic Systemscited by 2
  13. 2026Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressedcited by 2
  14. 2026Unlocking the Edge deployment and ondevice acceleration of multi-LoRA enabled one-for-all foundational LLMcited by 1
  15. 2024OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruningcited by 0
  16. 2024Personal Intelligence System UniLM: Hybrid On-Device Small Language Model and Server-Based Large Language Model for Malay Nusantaracited by 0
  17. 2024Responsible AI in NLP: GUS-Net Span-Level Bias Detection Dataset and Benchmark for Generalizations, Unfairness, and Stereotypescited by 0
  18. 2025SDMPrune: Self-Distillation MLP Pruning for Efficient Large Language Modelscited by 0
  19. 2025An Offline Mobile Conversational Agent for Mental Health Support: Learning from Emotional Dialogues and Psychological Texts with Student-Centered Evaluationcited by 0
  20. 2026Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigmcited by 0
  21. 2026MedRoute: RL-Based Dynamic Specialist Routing in Multi-Agent Medical Diagnosiscited by 0
  22. 2026BigMoMo: Efficient Inference of Large-Scale MoE with Speculative Decoding on Mobile Devicescited by 0