# Na Pr'atica, qual IA Entende o Direito? Um Estudo Experimental com IAs Generalistas e uma IA Jur'idica

arXiv: https://arxiv.org/abs/2510.18108 · HTML: https://mlsanity.com/paper/2510.18108/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 25 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 9 of 9)

- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [LLM-based NLG Evaluation: Current Status and Challenges](https://mlsanity.com/paper/2402.01383/) (2024)
- [JuDGE: Benchmarking Judgment Document Generation for Chinese Legal System](https://mlsanity.com/paper/2503.14258/) (2025)
- [Automatic Legal Writing Evaluation of LLMs](https://mlsanity.com/paper/2504.21202/) (2025)
- [An Evaluation Framework for Legal Document Summarization](https://mlsanity.com/paper/2205.08478/) (2022)
- [CaseGen: A Benchmark for Multi-Stage Legal Case Documents Generation](https://mlsanity.com/paper/2502.17943/) (2025)
- [Question-Answering Approach to Evaluating Legal Summaries](https://mlsanity.com/paper/2309.15016/) (2023)
- [LRAGE: Legal Retrieval Augmented Generation Evaluation Tool](https://mlsanity.com/paper/2504.01840/) (2025)

## Abstract (arXiv)

This study presents the Jusbrasil Study on the Use of General-Purpose AIs in Law, proposing an experimental evaluation protocol combining legal theory, such as material correctness, systematic coherence, and argumentative integrity, with empirical assessment by 48 legal professionals. Four systems (JusIA, ChatGPT Free, ChatGPT Pro, and Gemini) were tested in tasks simulating lawyers' daily work. JusIA, a domain-specialized model, consistently outperformed the general-purpose systems, showing that both domain specialization and a theoretically grounded evaluation are essential for reliable legal AI outputs.