# CalliRewrite: Recovering Handwriting Behaviors from Calligraphy Images without Supervision

arXiv: https://arxiv.org/abs/2405.15776 · HTML: https://mlsanity.com/paper/2405.15776/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 49 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [DG-Font: Deformable Generative Networks for Unsupervised Font Generation](https://mlsanity.com/paper/2104.03064/) (2021)
- [Auto-Encoder Guided GAN for Chinese Calligraphy Synthesis](https://mlsanity.com/paper/1706.08789/) (2017)
- [Learning to Use Chopsticks in Diverse Gripping Styles](https://mlsanity.com/paper/2205.14313/) (2022)
- [DeepVecFont-v2: Exploiting Transformers to Synthesize Vector Fonts with Higher Quality](https://mlsanity.com/paper/2303.14585/) (2023)
- [Content Masked Loss: Human-Like Brush Stroke Planning in a Reinforcement Learning Painting Agent](https://mlsanity.com/paper/2012.10043/) (2020)
- [PBCS : Efficient Exploration and Exploitation Using a Synergy between Reinforcement Learning and Motion Planning](https://mlsanity.com/paper/2004.11667/) (2020)
- [Pyramid Embedded Generative Adversarial Network for Automated Font Generation](https://mlsanity.com/paper/1811.08106/) (2018)
- [VecFontSDF: Learning to Reconstruct and Synthesize High-quality Vector Fonts via Signed Distance Functions](https://mlsanity.com/paper/2303.12675/) (2023)
- [Artistic Style in Robotic Painting; a Machine Learning Approach to Learning Brushstroke from Human Artists](https://mlsanity.com/paper/2007.03647/) (2020)
- [FontTransformer: Few-shot High-resolution Chinese Glyph Image Synthesis via Stacked Transformers](https://mlsanity.com/paper/2210.06301/) (2022)
- [Robot Calligraphy using Pseudospectral Optimal Control in Conjunction with a Novel Dynamic Brush Model](https://mlsanity.com/paper/2003.01565/) (2020)
- [Robot Navigation with Reinforcement Learned Path Generation and Fine-Tuned Motion Control](https://mlsanity.com/paper/2210.10639/) (2022)
- [CLAMGen: Closed-Loop Arm Motion Generation via Multi-view Vision-Based RL](https://mlsanity.com/paper/2103.13267/) (2021)
- [End-to-end Manipulator Calligraphy Planning via Variational Imitation Learning](https://mlsanity.com/paper/2304.02801/) (2023)

## Built on by (most cited first, 1 of 1)

- [Bridging Online and Offline Handwriting via Differentiable Physical Rendering](https://mlsanity.com/paper/2608.03198/) (2026)

## Abstract (arXiv)

Human-like planning skills and dexterous manipulation have long posed challenges in the fields of robotics and artificial intelligence (AI). The task of reinterpreting calligraphy presents a formidable challenge, as it involves the decomposition of strokes and dexterous utensil control. Previous efforts have primarily focused on supervised learning of a single instrument, limiting the performance of robots in the realm of cross-domain text replication. To address these challenges, we propose CalliRewrite: a coarse-to-fine approach for robot arms to discover and recover plausible writing orders from diverse calligraphy images without requiring labeled demonstrations. Our model achieves fine-grained control of various writing utensils. Specifically, an unsupervised image-to-sequence model decomposes a given calligraphy glyph to obtain a coarse stroke sequence. Using an RL algorithm, a simulated brush is fine-tuned to generate stylized trajectories for robotic arm control. Evaluation in simulation and physical robot scenarios reveals that our method successfully replicates unseen fonts and styles while achieving integrity in unknown characters.