# Passing the Brazilian OAB Exam: data preparation and some experiments

arXiv: https://arxiv.org/abs/1712.05128 · HTML: https://mlsanity.com/paper/1712.05128/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2017.

## Built on by (most cited first, 6 of 6)

- [Multilingual != Multicultural: Evaluating Gaps Between Multilingual Capabilities and Cultural Alignment in LLMs](https://mlsanity.com/paper/2502.16534/) (2025)
- [Tucano: Advancing Neural Text Generation for Portuguese](https://mlsanity.com/paper/2411.07854/) (2024)
- [LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domain](https://mlsanity.com/paper/2404.02127/) (2024)
- [PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese](https://mlsanity.com/paper/2511.17808/) (2025)
- [BLUEX Revisited: Enhancing Benchmark Coverage with Automatic Captioning](https://mlsanity.com/paper/2508.21294/) (2025)
- [BRoverbs -- Measuring how much LLMs understand Portuguese proverbs](https://mlsanity.com/paper/2509.08960/) (2025)

## Abstract (arXiv)

In Brazil, all legal professionals must demonstrate their knowledge of the law and its application by passing the OAB exams, the national bar exams. The OAB exams therefore provide an excellent benchmark for the performance of legal information systems since passing the exam would arguably signal that the system has acquired capacity of legal reasoning comparable to that of a human lawyer. This article describes the construction of a new data set and some preliminary experiments on it, treating the problem of finding the justification for the answers to questions. The results provide a baseline performance measure against which to evaluate future improvements. We discuss the reasons to the poor performance and propose next steps.