# Motion Switching with Sensory and Instruction Signals by designing Dynamical Systems using Deep Neural Network

arXiv: https://arxiv.org/abs/1712.05109 · HTML: https://mlsanity.com/paper/1712.05109/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Dec 2017.

## Built on by (most cited first, 5 of 5)

- [In-air Knotting of Rope using Dual-Arm Robot based on Deep Learning](https://mlsanity.com/paper/2103.09402/) (2021)
- [Embodying Pre-Trained Word Embeddings Through Robot Actions](https://mlsanity.com/paper/2104.08521/) (2021)
- [System 0/1/2/3: Quad-process theory for multi-timescale embodied collective cognitive systems](https://mlsanity.com/paper/2503.06138/) (2025)
- [Stable deep reinforcement learning method by predicting uncertainty in rewards as a subtask](https://mlsanity.com/paper/2101.06906/) (2021)
- [A Low-complexity Structured Neural Network to Realize States of Dynamical Systems](https://mlsanity.com/paper/2503.23697/) (2025)

## Abstract (arXiv)

To ensure that a robot is able to accomplish an extensive range of tasks, it is necessary to achieve a flexible combination of multiple behaviors. This is because the design of task motions suited to each situation would become increasingly difficult as the number of situations and the types of tasks performed by them increase. To handle the switching and combination of multiple behaviors, we propose a method to design dynamical systems based on point attractors that accept (i) "instruction signals" for instruction-driven switching. We incorporate the (ii) "instruction phase" to form a point attractor and divide the target task into multiple subtasks. By forming an instruction phase that consists of point attractors, the model embeds a subtask in the form of trajectory dynamics that can be manipulated using sensory and instruction signals. Our model comprises two deep neural networks: a convolutional autoencoder and a multiple time-scale recurrent neural network. In this study, we apply the proposed method to manipulate soft materials. To evaluate our model, we design a cloth-folding task that consists of four subtasks and three patterns of instruction signals, which indicate the direction of motion. The results depict that the robot can perform the required task by combining subtasks based on sensory and instruction signals. And, our model determined the relations among these signals using its internal dynamics.