# CrochetBench: Can Vision-Language Models Move from Describing to Doing in Crochet Domain?

arXiv: https://arxiv.org/abs/2511.09483 · HTML: https://mlsanity.com/paper/2511.09483/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 36 references that are in the map.
- First posted on arXiv 12 Nov 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 19 of 19)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [DeepSeek-VL: Towards Real-World Vision-Language Understanding](https://mlsanity.com/paper/2403.05525/) (2024)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation](https://mlsanity.com/paper/2010.16061/) (2020)
- [TextCaps: a Dataset for Image Captioning with Reading Comprehension](https://mlsanity.com/paper/2003.12462/) (2020)
- [Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food Images](https://mlsanity.com/paper/1810.06553/) (2018)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Why are Visually-Grounded Language Models Bad at Image Classification?](https://mlsanity.com/paper/2405.18415/) (2024)
- [Image-to-Markup Generation with Coarse-to-Fine Attention](https://mlsanity.com/paper/1609.04938/) (2016)
- [ChefFusion: Multimodal Foundation Model Integrating Recipe and Food Image Generation](https://mlsanity.com/paper/2409.12010/) (2024)
- [Neural Inverse Knitting: From Images to Manufacturing Instructions](https://mlsanity.com/paper/1902.02752/) (2019)
- [ChemHGNN: A Hierarchical Hypergraph Neural Network for Reaction Virtual Screening and Discovery](https://mlsanity.com/paper/2506.11041/) (2025)

## Built on by (most cited first, 2 of 2)

- [EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answering](https://mlsanity.com/paper/2604.05149/) (2026)
- [Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization](https://mlsanity.com/paper/2605.21801/) (2026)

## Abstract (arXiv)

While multimodal large language models can describe visual content, their ability to generate executable procedures remains underexplored. CrochetBench presented in this paper evaluates this shift from describing to doing through fine-grained procedural reasoning in crochet: models must recognize stitches, select structurally appropriate instructions, and generate compilable procedures. We adopt the CrochetPARADE DSL as our intermediate representation, enabling structural validation and functional evaluation via execution. The benchmark covers tasks including stitch classification, instruction grounding, and both natural language and image-to-DSL translation. Across all tasks, performance sharply decreases as the evaluation shifts from surface-level similarity to executable correctness, revealing limitations in long-range symbolic reasoning and 3D-aware procedural synthesis. Our proposed CrochetBench offers a new lens for assessing procedural competence in multimodal models and highlights the gap between surface-level understanding and executable precision in real-world creative domains. Code is available at https://github.com/Peiyu-Georgia-Li/crochetBench.