mlsanityData to 14 Oct 2026

Raising Bars, Not Parameters: LilMoo Compact Language Model for Hindi

arXiv 2603.03508 ↗First posted 3 Mar 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The dominance of large multilingual foundation models has widened linguistic inequalities in Natural Language Processing (NLP), often leaving low-resource languages underrepresented. This paper introduces LilMoo, a 0.6-billion-parameter Hindi language model trained entirely from scratch to address this gap. Unlike prior Hindi models that rely on continual pretraining from opaque multilingual foundations, LilMoo is developed through a fully transparent and reproducible pipeline optimized for limited compute environments. We construct a high-quality Hindi corpus (GigaLekh) filtered through both heuristic and learned (LLM-as-a-judge) methods, complemented by bilingual augmentation with curated English data. Using this dataset, we explore various training recipes for small-scale language models. Across comprehensive evaluation suites, LilMoo consistently outperforms comparably sized multilingual baselines such as Qwen2.5-0.5B and Qwen3-0.6B, demonstrating that well-designed language-specific pretraining can rival large multilingual models at the sub-billion-parameter range.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 65 of its 84 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

65 of 84 references.

References · 65

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2025Qwen3 Technical Reportcited by 8,820
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  10. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
38 more
  1. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  2. 2020GLU Variants Improve Transformercited by 1,804
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2019Root Mean Square Layer Normalizationcited by 1,480
  5. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  6. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  7. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 642
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  13. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  14. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
  15. 2023Scaling Data-Constrained Language Modelscited by 403
  16. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  17. 2019CamemBERT: a Tasty French Language Modelcited by 368
  18. 20242 OLMo 2 Furiouscited by 340
  19. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 335
  20. 2023A Cookbook of Self-Supervised Learningcited by 250
  21. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  22. 2023Continual Pre-training of Language Modelscited by 198
  23. 2018Measuring the Effects of Data Parallelism on Neural Network Trainingcited by 193
  24. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  25. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 174
  26. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  27. 2024Continual Learning with Pre-Trained Models: A Surveycited by 154
  28. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  29. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  30. 2025FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Languagecited by 143
  31. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  32. 2019Multilingual is not enough: BERT for Finnishcited by 119
  33. 2024OLMES: A Standard for Language Model Evaluationscited by 106
  34. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 104
  35. 2025Falcon-H1: A Family of Hybrid-Head Language Models Redefining Efficiency and Performancecited by 57
  36. 2023GlotLID: Language Identification for Low-Resource Languagescited by 39
  37. 2022L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languagescited by 35
  38. 2023When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languagescited by 32

50 most cited of 65 All 65 (JSON)

Built on by

Not cited yet.