# RoQLlama: A Lightweight Romanian Adapted Language Model

arXiv: https://arxiv.org/abs/2410.04269 · HTML: https://mlsanity.com/paper/2410.04269/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 5 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca](https://mlsanity.com/paper/2304.08177/) (2023)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [TableLlama: Towards Open Large Generalist Models for Tables](https://mlsanity.com/paper/2311.09206/) (2023)
- [IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding](https://mlsanity.com/paper/2009.05387/) (2020)
- [ORCA: A Challenging Benchmark for Arabic Language Understanding](https://mlsanity.com/paper/2212.10758/) (2022)
- [The birth of Romanian BERT](https://mlsanity.com/paper/2009.08712/) (2020)
- [Parameter-Efficient Fine-Tuning of LLaMA for the Clinical Domain](https://mlsanity.com/paper/2307.03042/) (2023)
- [ArchivalQA: A Large-scale Benchmark Dataset for Open Domain Question Answering over Historical News Collections](https://mlsanity.com/paper/2109.03438/) (2021)
- [A Continued Pretrained LLM Approach for Automatic Medical Note Generation](https://mlsanity.com/paper/2403.09057/) (2024)
- [LLaMAntino: LLaMA 2 Models for Effective Text Generation in Italian Language](https://mlsanity.com/paper/2312.09993/) (2023)
- [Teaching Llama a New Language Through Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2404.04042/) (2024)
- [GeoSQA: A Benchmark for Scenario-based Question Answering in the Geography Domain at High School Level](https://mlsanity.com/paper/1908.07855/) (2019)
- [MOROCO: The Moldavian and Romanian Dialectal Corpus](https://mlsanity.com/paper/1901.06543/) (2019)
- [SaRoCo: Detecting Satire in a Novel Romanian Corpus of News Articles](https://mlsanity.com/paper/2105.06456/) (2021)
- [ParaShoot: A Hebrew Question Answering Dataset](https://mlsanity.com/paper/2109.11314/) (2021)
- [LLaMandement: Large Language Models for Summarization of French Legislative Proposals](https://mlsanity.com/paper/2401.16182/) (2024)
- [Open Generative Large Language Models for Galician](https://mlsanity.com/paper/2406.13893/) (2024)

## Built on by (most cited first, 7 of 7)

- [GRAF: Graph Retrieval Augmented by Facts for Romanian Legal Multi-Choice Question Answering](https://mlsanity.com/paper/2412.04119/) (2024)
- [RoLargeSum: A Large Dialect-Aware Romanian News Dataset for Summary, Headline, and Keyword Generation](https://mlsanity.com/paper/2412.11317/) (2024)
- [A Large-Scale Benchmark for Evaluating Large Language Models on Medical Question Answering in Romanian](https://mlsanity.com/paper/2508.16390/) (2025)
- [MoRoVoc: A Large Dataset for Geographical Variation Identification of the Spoken Romanian Language](https://mlsanity.com/paper/2509.16781/) (2025)
- [RoD-TAL: A Benchmark for Answering Questions in Romanian Driving License Exams](https://mlsanity.com/paper/2507.19666/) (2025)
- [RoBiologyDataChoiceQA: A Romanian Dataset for improving Biology understanding of Large Language Models](https://mlsanity.com/paper/2509.25813/) (2025)
- [PolERo: Studying Political Evasion in Romanian](https://mlsanity.com/paper/2609.02391/) (2026)

## Abstract (arXiv)

The remarkable achievements obtained by open-source large language models (LLMs) in recent years have predominantly been concentrated on tasks involving the English language. In this paper, we aim to advance the performance of Llama2 models on Romanian tasks. We tackle the problem of reduced computing resources by using QLoRA for training. We release RoQLlama-7b, a quantized LLM, which shows equal or improved results compared to its full-sized counterpart when tested on seven Romanian downstream tasks in the zero-shot setup. Also, it consistently achieves higher average scores across all few-shot prompts. Additionally, we introduce a novel Romanian dataset, namely RoMedQA, which contains single-choice medical questions in Romanian.