# The Large Language Model GreekLegalRoBERTa

arXiv: https://arxiv.org/abs/2410.12852 · HTML: https://mlsanity.com/paper/2410.12852/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Builds on (most cited first, 25 of 25)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Nomic Embed: Training a Reproducible Long Context Text Embedder](https://mlsanity.com/paper/2402.01613/) (2024)
- [Arcee's MergeKit: A Toolkit for Merging Large Language Models](https://mlsanity.com/paper/2403.13257/) (2024)
- [A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages](https://mlsanity.com/paper/2006.06202/) (2020)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development](https://mlsanity.com/paper/2305.07507/) (2023)
- [PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation](https://mlsanity.com/paper/2004.07159/) (2020)
- [GREEK-BERT: The Greeks visiting Sesame Street](https://mlsanity.com/paper/2008.12014/) (2020)
- [Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning](https://mlsanity.com/paper/1705.00557/) (2017)
- [Multi-granular Legal Topic Classification on Greek Legislation](https://mlsanity.com/paper/2109.15298/) (2021)

## Abstract (arXiv)

We develop four versions of GreekLegalRoBERTa, which are four large language models trained on Greek legal and nonlegal text. We show that our models surpass the performance of GreekLegalBERT, Greek- LegalBERT-v2, and GreekBERT in two tasks involving Greek legal documents: named entity recognition and multi-class legal topic classification. We view our work as a contribution to the study of domain-specific NLP tasks in low-resource languages, like Greek, using modern NLP techniques and methodologies.