# Robot Cooking with Stir-fry: Bimanual Non-prehensile Manipulation of Semi-fluid Objects

arXiv: https://arxiv.org/abs/2205.05960 · HTML: https://mlsanity.com/paper/2205.05960/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 27 references that are in the map.
- First posted on arXiv 12 May 2022.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Graph Transformer Networks](https://mlsanity.com/paper/1911.06455/) (2019)
- [SR-LSTM: State Refinement for LSTM towards Pedestrian Trajectory Prediction](https://mlsanity.com/paper/1903.02793/) (2019)
- [Graph Transformer for Graph-to-Sequence Learning](https://mlsanity.com/paper/1911.07470/) (2019)
- [Scheduled Sampling for Transformers](https://mlsanity.com/paper/1906.07651/) (2019)
- [Survey on Deep Neural Networks in Speech and Vision Systems](https://mlsanity.com/paper/1908.07656/) (2019)
- [Bayesian Gaussian mixture model for robotic policy imitation](https://mlsanity.com/paper/1904.10716/) (2019)
- [Compliant Movement Primitives in a Bimanual Setting](https://mlsanity.com/paper/1707.04629/) (2017)

## Built on by (most cited first, 25 of 25)

- [PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks](https://mlsanity.com/paper/2407.00278/) (2024)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulation](https://mlsanity.com/paper/2207.01971/) (2022)
- [Bi-KVIL: Keypoints-based Visual Imitation Learning of Bimanual Manipulation Tasks](https://mlsanity.com/paper/2403.03270/) (2024)
- [Multimedia Generative Script Learning for Task Planning](https://mlsanity.com/paper/2208.12306/) (2022)
- [Exploiting Transformer in Sparse Reward Reinforcement Learning for Interpretable Temporal Logic Motion Planning](https://mlsanity.com/paper/2209.13220/) (2022)
- [SoftGPT: Learn Goal-oriented Soft Object Manipulation Skills by Generative Pre-trained Heterogeneous Graph Transformer](https://mlsanity.com/paper/2306.12677/) (2023)
- [BiKC: Keypose-Conditioned Consistency Policy for Bimanual Robotic Manipulation](https://mlsanity.com/paper/2406.10093/) (2024)
- [Bi-LAT: Bilateral Control-Based Imitation Learning via Natural Language and Action Chunking with Transformers](https://mlsanity.com/paper/2504.01301/) (2025)
- [BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly](https://mlsanity.com/paper/2506.06221/) (2025)
- [ReVoLT: Relational Reasoning and Voronoi Local Graph Planning for Target-driven Navigation](https://mlsanity.com/paper/2301.02382/) (2023)
- [BiRP: Learning Robot Generalized Bimanual Coordination using Relative Parameterization Method on Human Demonstration](https://mlsanity.com/paper/2307.05933/) (2023)
- [ManiDP: Manipulability-Aware Diffusion Policy for Posture-Dependent Bimanual Manipulation](https://mlsanity.com/paper/2510.23016/) (2025)
- [Interactively Robot Action Planning with Uncertainty Analysis and Active Questioning by Large Language Model](https://mlsanity.com/paper/2308.15684/) (2023)
- [Fusion Dynamical Systems with Machine Learning in Imitation Learning: A Comprehensive Overview](https://mlsanity.com/paper/2403.19916/) (2024)
- [Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress](https://mlsanity.com/paper/2509.24129/) (2025)
- [MILE: A Mechanically Isomorphic Hand Exoskeleton and Visuotactile Robotic Hand for Data Collection in Dexterous Manipulation](https://mlsanity.com/paper/2512.00324/) (2025)
- [Learning Multimodal Attention for Manipulating Deformable Objects with Changing States](https://mlsanity.com/paper/2309.14837/) (2023)
- [Understanding Particles From Video: Property Estimation of Granular Materials via Visuo-Haptic Learning](https://mlsanity.com/paper/2412.02119/) (2024)
- [Human-Humanoid Robots Cross-Embodiment Behavior-Skill Transfer Using Decomposed Adversarial Learning from Demonstration](https://mlsanity.com/paper/2412.15166/) (2024)
- [SCOOP'D: Learning Mixed-Liquid-Solid Scooping via Sim2Real Generative Policy](https://mlsanity.com/paper/2510.11566/) (2025)
- [Polyphony: Diffusion-based Dual-Hand Action Segmentation with Alternating Vision Transformer and Semantic Conditioning](https://mlsanity.com/paper/2605.31115/) (2026)
- [Learning Goal-oriented Bimanual Dough Rolling Using Dynamic Heterogeneous Graph Based on Human Demonstration](https://mlsanity.com/paper/2410.22355/) (2024)
- [AsymDex: Asymmetry and Relative Coordinates for RL-based Bimanual Dexterity](https://mlsanity.com/paper/2411.13020/) (2024)
- [Leveraging Foundation Models To learn the shape of semi-fluid deformable objects](https://mlsanity.com/paper/2411.16802/) (2024)

## Abstract (arXiv)

This letter describes an approach to achieve well-known Chinese cooking art stir-fry on a bimanual robot system. Stir-fry requires a sequence of highly dynamic coordinated movements, which is usually difficult to learn for a chef, let alone transfer to robots. In this letter, we define a canonical stir-fry movement, and then propose a decoupled framework for learning this deformable object manipulation from human demonstration. First, the dual arms of the robot are decoupled into different roles (a leader and follower) and learned with classical and neural network-based methods separately, then the bimanual task is transformed into a coordination problem. To obtain general bimanual coordination, we secondly propose a Graph and Transformer based model -- Structured-Transformer, to capture the spatio-temporal relationship between dual-arm movements. Finally, by adding visual feedback of content deformation, our framework can adjust the movements automatically to achieve the desired stir-fry effect. We verify the framework by a simulator and deploy it on a real bimanual Panda robot system. The experimental results validate our framework can realize the bimanual robot stir-fry motion and have the potential to extend to other deformable objects with bimanual coordination.