# Large Malaysian Language Model Based on Mistral for Enhanced Local Language Understanding

arXiv: https://arxiv.org/abs/2401.13565 · HTML: https://mlsanity.com/paper/2401.13565/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 23 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Builds on (most cited first, 4 of 4)

- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [WizardCoder: Empowering Code Large Language Models with Evol-Instruct](https://mlsanity.com/paper/2306.08568/) (2023)

## Built on by (most cited first, 4 of 4)

- [Can LLM Generate Culturally Relevant Commonsense QA Data? Case Study in Indonesian and Sundanese](https://mlsanity.com/paper/2402.17302/) (2024)
- [Multi-Lingual Malaysian Embedding: Leveraging Large Language Models for Semantic Representations](https://mlsanity.com/paper/2402.03053/) (2024)
- [Soro: A Lightweight Foundation Model and Chatbot for Tajik](https://mlsanity.com/paper/2605.27379/) (2026)
- [MaLLaM -- Malaysia Large Language Model](https://mlsanity.com/paper/2401.14680/) (2024)

## Abstract (arXiv)

In this paper, we present significant advancements in the pretraining of Mistral 7B, a large-scale language model, using a dataset of 32.6 GB, equivalent to 1.1 billion tokens. We explore the impact of extending the context length, releasing models with context lengths of 4096 and 32768 tokens, and further refining performance with a specialized 16384 context length instruction-tuned model, we called it Malaysian Mistral. Our experiments demonstrate the efficacy of continue pretraining and the influence of extended context lengths on Mistral 7B's language understanding capabilities. Additionally, we release a model specifically tuned with a 16384 context length instruction, showcasing its potential for capturing nuanced language intricacies. Furthermore, our research contributes to the benchmarking of Malaysian Mistral against prominent language models, including ChatGPT3.5 and Claude 2. We present compelling results indicating Malaysian Mistral's superior performance on Tatabahasa (Malay grammar) test set, particularly when fine-tuned with instructions. All models released at https://huggingface.co/collections/mesolitica/malaysian-mistral-7b-6528f2ec825f4bba46c1700c