Ayn: A Tiny yet Competitive Indian Legal Language Model Pretrained from Scratch
Not on the map. Why?
Abstract (arXiv)
Decoder-only Large Language Models (LLMs) are currently the model of choice for many Natural Language Processing (NLP) applications. Through instruction fine-tuning and prompting approaches, such LLMs have been efficiently used to solve both general and domain-specific tasks. However, they are costly to train and, to a certain extent, costly to use as well, and one can wonder whether LLMs can be replaced by domain-specific Tiny Language Models (TLMs), which typically contain less than 100M parameters. We address this question in this study by comparing the performance of an 88M TLM pretrained from scratch for 185 A100 hours on a specific domain with a domain-specific tokenizer (here, the Indian legal domain) with LLMs of various sizes between 1B and 8B for solving domain-specific tasks. We show in particular that our legal TLM, Ayn, can indeed outperform LLMs up to 80 times larger on the legal case judgment prediction task, rival LLMs up to 30 times larger on the summarization task, and still be competitive with these larger LLMs on general tasks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 66 references that are in the map. First posted on arXiv 20 Mar 2024.
Builds on
29 of 66 references.
References · 29
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2020GLU Variants Improve Transformercited by 1,804
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
17 more
- 2019Root Mean Square Layer Normalizationcited by 1,480
- 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
- 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 372
- 2016Using the Output Embedding to Improve Language Modelscited by 272
- 2019Neural Legal Judgment Prediction in Englishcited by 209
- 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
- 2024Towards Measuring and Modeling "Culture" in LLMs: A Surveycited by 167
- 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
- 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 142
- 2023f-Divergence Minimization for Sequence-Level Knowledge Distillationcited by 114
- 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
- 2021ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanationcited by 78
- 2022PODA: Prompt-driven Zero-shot Domain Adaptationcited by 60
- 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
- 2021Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domainscited by 36
- 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
- 2024Exploring Large Language Models and Hierarchical Frameworks for Classification of Large Unstructured Legal Documentscited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.