Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance
Not on the map. Why?
Abstract (arXiv)
Small Language Models (SLMs) offer efficient alternatives to LLMs for specific domains. The 2023 TinyStories study developed an English dataset that allows SLMs with 1 to 10 million parameters to produce coherent outputs. Our research expands this framework by translating the original dataset into Indian languages and creating synthetic data using LLMs. We focus on Hindi, Marathi, and Bengali, evaluating SLMs for regional language processing and understanding linguistic complexity. We show that SLMs efficiently process regional languages with significantly fewer parameters than LLMs, providing a complementary framework for ``inference based evaluation" of tokenization strategies and linguistic complexity. Our analysis shows that language-specific tokenizers outperform general-purpose ones for Indian languages. Empirical validations, supported by information-theoretic and morphological analyses, provides fundamental understanding behind the better performance of Hindi models over Marathi and Bengali. Additionally, we show that synthetic datasets outperform translated content for training SLMs. Correlation analyses reveal cross-linguistic patterns and language-specific relationships between creativity, grammatical precision, and narrative completeness. These findings advance both the practical application of SLMs to underserved languages and our theoretical understanding of neural language development.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 24 of its 50 references that are in the map. First posted on arXiv 7 Apr 2025.
Builds on
24 of 50 references.
References · 24
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
- 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
- 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
- 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
- 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
- 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 176
- 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
- 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
- 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
12 more
- 2023Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpuscited by 62
- 2024Why do language models perform worse for morphologically complex languages?cited by 62
- 2023Tokenization and the Noiseless Channelcited by 59
- 2022Democratizing Neural Machine Translation with OPUS-MTcited by 58
- 2019The Effect of Translationese in Machine Translation Test Setscited by 49
- 2021IndicBART: A Pre-trained Model for Indic Natural Language Generationcited by 39
- 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
- 2021Inferring the Reader: Guiding Automated Story Generation with Commonsense Reasoningcited by 29
- 2021Guiding Neural Story Generation with Reader Modelscited by 12
- 2024Emergent Abilities in Reduced-Scale Generative Language Modelscited by 7
- 2024Pretraining Language Models Using Translationesecited by 6
- 2024Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysiscited by 5
Built on by
Citing papers per year: 2025: 1.
1 paper.