# DiLogics: Creating Web Automation Programs With Diverse Logics

arXiv: https://arxiv.org/abs/2308.05828 · HTML: https://mlsanity.com/paper/2308.05828/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 61 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [Neuro-Symbolic Program Synthesis](https://mlsanity.com/paper/1611.01855/) (2016)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations](https://mlsanity.com/paper/1909.00031/) (2019)
- [Mapping Natural Language Commands to Web Elements](https://mlsanity.com/paper/1808.09132/) (2018)
- [Scout: Rapid Exploration of Interface Layout Alternatives through High-Level Design Constraints](https://mlsanity.com/paper/2001.05424/) (2020)
- [Web Question Answering with Neurosymbolic Program Synthesis](https://mlsanity.com/paper/2104.07162/) (2021)
- [WebRobot: Web Robotic Process Automation using Interactive Programming-by-Demonstration](https://mlsanity.com/paper/2203.09993/) (2022)
- [VASTA: A Vision and Language-assisted Smartphone Task Automation System](https://mlsanity.com/paper/1911.01474/) (2019)

## Built on by (most cited first, 5 of 5)

- [AI2T: Building Trustable AI Tutors by Interactively Teaching a Self-Aware Learning Agent](https://mlsanity.com/paper/2411.17924/) (2024)
- [Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable Webpages](https://mlsanity.com/paper/2505.22831/) (2025)
- [ClearFairy: Capturing Creative Workflows through Decision Structuring, In-Situ Questioning, and Rationale Inference](https://mlsanity.com/paper/2509.14537/) (2025)
- [Efficient Bottom-Up Synthesis for Programs with Local Variables](https://mlsanity.com/paper/2311.03705/) (2023)
- [Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations](https://mlsanity.com/paper/2509.24250/) (2025)

## Abstract (arXiv)

Knowledge workers frequently encounter repetitive web data entry tasks, like updating records or placing orders. Web automation increases productivity, but translating tasks to web actions accurately and extending to new specifications is challenging. Existing tools can automate tasks that perform the same logical trace of UI actions (e.g., input text in each field in order), but do not support tasks requiring different executions based on varied input conditions. We present DiLogics, a programming-by-demonstration system that utilizes NLP to assist users in creating web automation programs that handle diverse specifications. DiLogics first semantically segments input data to structured task steps. By recording user demonstrations for each step, DiLogics generalizes the web macros to novel but semantically similar task requirements. Our evaluation showed that non-experts can effectively use DiLogics to create automation programs that fulfill diverse input instructions. DiLogics provides an efficient, intuitive, and expressive method for developing web automation programs satisfying diverse specifications.