mlsanityData to 14 Oct 2026

Merino: Entropy-driven Design for Generative Language Models on IoT Devices

arXiv 2403.07921 ↗First posted 28 Feb 2024cs.LG cs.AI cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Generative Large Language Models (LLMs) stand as a revolutionary advancement in the modern era of artificial intelligence (AI). However, scaling down LLMs for resource-constrained hardware, such as Internet-of-Things (IoT) devices requires non-trivial efforts and domain knowledge. In this paper, we propose a novel information-entropy framework for designing mobile-friendly generative language models. The whole design procedure involves solving a mathematical programming (MP) problem, which can be done on the CPU within minutes, making it nearly zero-cost. We evaluate our designed models, termed MeRino, across fourteen NLP downstream tasks, showing their competitive performance against the state-of-the-art autoregressive transformer models under the mobile setting. Notably, MeRino achieves similar or better performance on both language modeling and zero-shot learning tasks, compared to the 350M parameter OPT while being 4.9x faster on NVIDIA Jetson Nano with 5.5x reduction in model size.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 43 of its 62 references that are in the map. First posted on arXiv 28 Feb 2024.

Builds on

43 of 62 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  12. 2016Pointer Sentinel Mixture Modelscited by 3,096
31 more
  1. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  7. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  8. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  9. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  11. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  12. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  13. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  14. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  15. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  16. 2020ReZero is All You Need: Fast Convergence at Large Depthcited by 264
  17. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  18. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  19. 2021The Principles of Deep Learning Theorycited by 173
  20. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  21. 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
  22. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  23. 2021ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reductioncited by 84
  24. 2019Improving Deep Transformer with Depth-Scaled Initialization and Merged Attentioncited by 59
  25. 2021NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Searchcited by 56
  26. 2020Evaluating Efficient Performance Estimators of Neural Architecturescited by 55
  27. 2022Training-free Transformer Architecture Searchcited by 33
  28. 2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
  29. 2021A Short Study on Compressing Decoder-Based Language Modelscited by 16
  30. 2021MAE-DET: Revisiting Maximum Entropy Principle in Zero-Shot NAS for Efficient Object Detectioncited by 16
  31. 2023DeepMAD: Mathematical Architecture Design for Deep Convolutional Neural Networkcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  2. 2026TIDE: Efficient and Lossless MoE Diffusion LLM Inference with I/O-aware Expert Offloadcited by 2
  3. 2026MILO: Efficient Many-shot In-Context Learning with Block-wise Low-rank Compressioncited by 0