# NL2SHACL-Bench: A Benchmark Suite for Natural Language to SHACL Translation

arXiv: https://arxiv.org/abs/2608.07530 · HTML: https://mlsanity.com/paper/2608.07530/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 19 references that are in the map.
- First posted on arXiv 24 Jul 2026.

## Builds on (most cited first, 5 of 5)

- [Common Foundations for SHACL, ShEx, and PG-Schema](https://mlsanity.com/paper/2502.01295/) (2025)
- [Schema Generation for Large Knowledge Graphs Using Large Language Models](https://mlsanity.com/paper/2506.04512/) (2025)
- [SHACL2FOL: An FOL Toolkit for SHACL Decision Problems](https://mlsanity.com/paper/2406.08018/) (2024)
- [Automated Validation of Textual Constraints Against AutomationML via LLMs and SHACL](https://mlsanity.com/paper/2506.10678/) (2025)
- [RDFGraphGen: An RDF Graph Generator based on SHACL Shapes](https://mlsanity.com/paper/2407.17941/) (2024)

## Abstract (arXiv)

SHACL is a core technology for validating the conformance of RDF knowledge graphs (KGs). Yet, authoring SHACL shapes requires technical expertise that most domain experts lack. Translating natural language requirements into SHACL (NL2SHACL) would lower this barrier. However, there is no dedicated benchmark for NL2SHACL, and evaluating generated shapes requires methods beyond string comparison, as semantically equivalent shapes can differ in serialisation and structure. To tackle these challenges, we present NL2SHACL-Bench, a benchmark suite for natural language to SHACL translation. Using NL2SHACL-Bench, we evaluate four state-of-the-art large language models (LLMs) for this task. Our results show that current LLMs are highly capable of generating syntactically valid SHACL, but still struggle to produce semantically equivalent constraints for complex logical and structural patterns. This indicates that NL2SHACL-Bench provides a meaningful basis for measuring advances in the NL2SHACL state of the art.