mlsanityData to 7 Oct 2026

End-to-end Manipulator Calligraphy Planning via Variational Imitation Learning

arXiv 2304.02801 ↗First posted 6 Apr 2023cs.RO cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Planning from demonstrations has shown promising results with the advances of deep neural networks. One of the most popular real-world applications is automated handwriting using a robotic manipulator. Classically it is simplified as a two-dimension problem. This representation is suitable for elementary drawings, but it is not sufficient for Japanese calligraphy or complex work of art where the orientation of a pen is part of the user expression. In this study, we focus on automated planning of Japanese calligraphy using a three-dimension representation of the trajectory as well as the rotation of the pen tip, and propose a novel deep imitation learning neural network that learns from expert demonstrations through a combination of images and pose data. The network consists of a combination of variational auto-encoder, bi-directional LSTM, and Multi-Layer Perceptron (MLP). Experiments are conducted in a progressive way, and results demonstrate that the proposed approach is successful in completion of tasks for real-world robots, overcoming the distribution shift problem in imitation learning. The source code and dataset will be public.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 12 of its 23 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

12 of 23 references.

References · 12

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  6. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  7. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  8. 2018Object Detection with Deep Learning: A Reviewcited by 524
  9. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  10. 2016Modelling Radiological Language with Bidirectional Long Short-Term Memory Networkscited by 9
  11. 2021In-air Knotting of Rope using Dual-Arm Robot based on Deep Learningcited by 7
  12. 2020A Generalized Robotic Handwriting Learning System based on Dynamic Movement Primitives (DMPs)cited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024CalliRewrite: Recovering Handwriting Behaviors from Calligraphy Images without Supervisioncited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2019A Linearly Constrained Nonparametric Framework for Imitation Learning
  2. 2026Rapid Learning of Dexterous In-Hand Pen Writing through Real-Time Jacobian Estimation ↗
  3. 2025Non-expert to Expert Motion Translation Using Generative Adversarial Networks ↗
  4. 2025Latent Diffusion Planning for Imitation Learning
  5. 2024Learning from Demonstration with Implicit Nonlinear Dynamics Models
  6. 2023Imitating Task and Motion Planning with Visuomotor Transformers
  7. 2024Conditional Variational Auto Encoder Based Dynamic Motion for Multi-task Imitation Learning
  8. 2023Deep Learning for Koopman-based Dynamic Movement Primitives