# JU'A -- A Benchmark for Information Retrieval in Brazilian Legal Text Collections

arXiv: https://arxiv.org/abs/2604.06098 · HTML: https://mlsanity.com/paper/2604.06098/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 29 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [Explicitly Integrating Judgment Prediction with Legal Document Retrieval: A Law-Guided Generative Approach](https://mlsanity.com/paper/2312.09591/) (2023)
- [Systematic Evaluation of Neural Retrieval Models on the Touch'e 2020 Argument Retrieval Subset of BEIR](https://mlsanity.com/paper/2407.07790/) (2024)
- [Quati: A Brazilian Portuguese Information Retrieval Dataset from Native Speakers](https://mlsanity.com/paper/2404.06976/) (2024)
- [JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgments](https://mlsanity.com/paper/2503.08379/) (2025)
- [BR-TaxQA-R: A Dataset for Question Answering with References for Brazilian Personal Income Tax Law, including case law](https://mlsanity.com/paper/2505.15916/) (2025)

## Abstract (arXiv)

Legal information retrieval in Portuguese remains difficult to evaluate systematically because available datasets differ widely in document type, query style, and relevance definition. We present JU'A, a public benchmark for Brazilian legal retrieval designed to support more reproducible and comparable evaluation across heterogeneous legal collections. More broadly, JU'A is intended not only as a benchmark, but as a continuous evaluation infrastructure for Brazilian legal IR, combining shared protocols, common ranking metrics, fixed splits when applicable, and a public leaderboard. The benchmark covers jurisprudence retrieval as well as broader legislative, regulatory, and question-driven legal search. We evaluate lexical, dense, and BM25-based reranking pipelines, including a domain-adapted Qwen embedding model fine-tuned on JU'A-aligned supervision. Results show that the benchmark is sufficiently heterogeneous to distinguish retrieval paradigms and reveal substantial cross-dataset trade-offs. Domain adaptation yields its clearest gains on the supervision-aligned JU'A-Juris subset, while BM25 remains highly competitive on other collections, especially in settings with strong lexical and institutional phrasing cues. Overall, JU'A provides a practical evaluation framework for studying legal retrieval across multiple Brazilian legal domains under a common benchmark design.