# NL2Scratch: An Executable Benchmark and Evaluation for Block-Based Programming

arXiv: https://arxiv.org/abs/2606.22061 · HTML: https://mlsanity.com/paper/2606.22061/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 20 Jun 2026.

## Builds on (most cited first, 15 of 15)

- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study](https://mlsanity.com/paper/2404.17136/) (2024)
- [ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12](https://mlsanity.com/paper/2402.04975/) (2024)
- [LitterBox: A Linter for Scratch Programs](https://mlsanity.com/paper/2102.07440/) (2021)
- [Scratch Copilot: Supporting Youth Creative Coding with AI](https://mlsanity.com/paper/2505.03867/) (2025)
- [ViScratch: Using Large Language Models and Gameplay Videos for Automated Feedback in Scratch](https://mlsanity.com/paper/2509.11065/) (2025)
- [LitterBox+: An Extensible Framework for LLM-enhanced Scratch Static Code Analysis](https://mlsanity.com/paper/2509.12021/) (2025)
- [ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming](https://mlsanity.com/paper/2602.00757/) (2026)
- [On the Applicability of Language Models to Block-Based Programs](https://mlsanity.com/paper/2302.03927/) (2023)
- [Automatically Generating Questions About Scratch Programs](https://mlsanity.com/paper/2510.11658/) (2025)

## Abstract (arXiv)

Block-based programming environments such as Scratch are widely used in early programming education, yet natural-language-to-code (NL2Code) research has focused primarily on text-based languages. Scratch programs are event-driven, visually compositional, and distributed across concurrent scripts, making conventional NL2Code assumptions and evaluation insufficient. We introduce NL2Scratch, an executable benchmark for natural-language-to-Scratch generation comprising 311,648 parser-valid NL--program pairs, whose program side is extracted from real Scratch projects and paired with semantically aligned NL descriptions. For reliable evaluation beyond surface overlap, we propose Semantic Alignment Consistency (SAC), an interpretable slot-level metric for measuring semantic agreement between descriptions and programs. With SAC, we construct a semantically validated pool of 23,594 examples, and a slot-balanced 800 diagnostic benchmark. Experiments across instruction-tuned and fine-tuned LLMs reveal a notable gap between lexical similarity and semantic alignment: models achieving token-level F1 above 0.93 often fail to attain perfect SAC, particularly on longer examples. Errors concentrate on operational slots like actions, conditions, and numeric arguments, exposing failure modes largely invisible under conventional metrics.