# Automatic code generation from sketches of mobile applications in end-user development using Deep Learning

arXiv: https://arxiv.org/abs/2103.05704 · HTML: https://mlsanity.com/paper/2103.05704/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 39 references that are in the map.
- First posted on arXiv 9 Mar 2021.

## Builds on (most cited first, 5 of 5)

- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps](https://mlsanity.com/paper/1802.02312/) (2018)
- [Sketch2Code: Transformation of Sketches to UI in Real-time Using Deep Neural Network](https://mlsanity.com/paper/1910.08930/) (2019)
- [Sketch2code: Generating a website from a paper mockup](https://mlsanity.com/paper/1905.13750/) (2019)
- [BlackBox Toolkit: Intelligent Assistance to UI Design](https://mlsanity.com/paper/2004.01949/) (2020)

## Built on by (most cited first, 2 of 2)

- [Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping](https://mlsanity.com/paper/2410.16232/) (2024)
- [Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation](https://mlsanity.com/paper/2510.27632/) (2025)

## Abstract (arXiv)

A common need for mobile application development by end-users or in computing education is to transform a sketch of a user interface into wireframe code using App Inventor, a popular block-based programming environment. As this task is challenging and time-consuming, we present the Sketch2aia approach that automates this process. Sketch2aia employs deep learning to detect the most frequent user interface components and their position on a hand-drawn sketch creating an intermediate representation of the user interface and then automatically generates the App Inventor code of the wireframe. The approach achieves an average user interface component classification accuracy of 87,72% and results of a preliminary user evaluation indicate that it generates wireframes that closely mirror the sketches in terms of visual similarity. The approach has been implemented as a web tool and can be used to support the end-user development of mobile applications effectively and efficiently as well as the teaching of user interface design in K-12.