# From Legal Text to Executable Decision Models: Evaluating Structured Representations for Legal Decision Model Generation

arXiv: https://arxiv.org/abs/2604.17153 · HTML: https://mlsanity.com/paper/2604.17153/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 26 references that are in the map.
- First posted on arXiv 18 Apr 2026.

## Builds on (most cited first, 4 of 4)

- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [From Text to Structure: Using Large Language Models to Support the Development of Legal Expert Systems](https://mlsanity.com/paper/2311.04911/) (2023)
- [The Impact of Prompt Programming on Function-Level Code Generation](https://mlsanity.com/paper/2412.20545/) (2024)
- [Using LLMs to Discover Legal Factors](https://mlsanity.com/paper/2410.07504/) (2024)

## Built on by (most cited first, 1 of 1)

- [By Their Fruits You Will Know Them: Comparing Formalizations of Law by the Decisions They Encode](https://mlsanity.com/paper/2605.25186/) (2026)

## Abstract (arXiv)

Transforming legal text into executable decision logic is a longstanding challenge in legal informatics. With the rise of LLMs, this task has gained renewed interest, but remains challenging due to requiring extensive manual coding and evaluation. We use a unique real-world dataset that pairs production-grade decision models with legal text from the Dutch Environment and Planning Act. These models power the Omgevingsloket government platform, where citizens check permit requirements for environmental activities. We study whether intermediate structured representations can improve LLM-based generation of executable decision models from legal text. We compare four input conditions: raw legal text, text enriched with semantic role labels, text enriched with input and output constraints, and text enriched with both. We evaluate along two dimensions: structural evaluation, through similarity to gold decision models with graph kernels and graphs' descriptive statistics, and outcome evaluation, through functional equivalence by executing models on pre-configured test scenarios. Our findings show that I/O constraints provide the dominant improvement (+37-54% similarity over baseline), while semantic role labels show modest improvements. Outcome evaluation shows that generated models match the gold standard on 51-53% of test scenarios, even though generated models are typically smaller and simpler. We find LLMs eliminate redundant pass-through logic that comprises up to 45-55% of nodes. Importantly, structural similarity and outcome equivalence are complementary: structural similarity does not guarantee outcome equivalence, and vice versa. To facilitate reproducibility, we publicly release our dataset of 95 production decision models with associated legal text and all experimental code.