# Benchmarking Multimodal Regex Synthesis with Complex Structures

arXiv: https://arxiv.org/abs/2005.00663 · HTML: https://mlsanity.com/paper/2005.00663/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2020.

## Built on by (most cited first, 10 of 10)

- [SatLM: Satisfiability-Aided Language Models Using Declarative Prompting](https://mlsanity.com/paper/2305.09656/) (2023)
- [RnG-KBQA: Generation Augmented Iterative Ranking for Knowledge Base Question Answering](https://mlsanity.com/paper/2109.08678/) (2021)
- [Grammar Prompting for Domain-Specific Language Generation with Large Language Models](https://mlsanity.com/paper/2305.19234/) (2023)
- [Leveraging Language to Learn Program Abstractions and Search Heuristics](https://mlsanity.com/paper/2106.11053/) (2021)
- [Communicating Natural Programs to Humans and Machines](https://mlsanity.com/paper/2106.07824/) (2021)
- [Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-SQL](https://mlsanity.com/paper/2205.12422/) (2022)
- [InfeRE: Step-by-Step Regex Generation via Chain of Inference](https://mlsanity.com/paper/2308.04041/) (2023)
- [Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks](https://mlsanity.com/paper/2410.08437/) (2024)
- [ObSynth: An Interactive Synthesis System for Generating Object Models from Natural Language Specifications](https://mlsanity.com/paper/2210.11468/) (2022)
- [Predicting Performance of Symbolic and Prompt Programs with Examples](https://mlsanity.com/paper/2605.21515/) (2026)

## Abstract (arXiv)

Existing datasets for regular expression (regex) generation from natural language are limited in complexity; compared to regex tasks that users post on StackOverflow, the regexes in these datasets are simple, and the language used to describe them is not diverse. We introduce StructuredRegex, a new regex synthesis dataset differing from prior ones in three aspects. First, to obtain structurally complex and realistic regexes, we generate the regexes using a probabilistic grammar with pre-defined macros observed from real-world StackOverflow posts. Second, to obtain linguistically diverse natural language descriptions, we show crowdworkers abstract depictions of the underlying regex and ask them to describe the pattern they see, rather than having them paraphrase synthetic language. Third, we augment each regex example with a collection of strings that are and are not matched by the ground truth regex, similar to how real users give examples. Our quantitative and qualitative analysis demonstrates the advantages of StructuredRegex over prior datasets. Further experimental results using various multimodal synthesis techniques highlight the challenge presented by our dataset, including non-local constraints and multi-modal inputs.