# V2CNet: A Deep Learning Framework to Translate Videos to Commands for Robotic Manipulation

arXiv: https://arxiv.org/abs/1903.10869 · HTML: https://mlsanity.com/paper/1903.10869/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Mar 2019.

## Built on by (most cited first, 5 of 5)

- [Coarse-to-Fine Reasoning for Visual Question Answering](https://mlsanity.com/paper/2110.02526/) (2021)
- [Global-Local Attention for Emotion Recognition](https://mlsanity.com/paper/2111.04129/) (2021)
- [Graph-based Person Signature for Person Re-Identifications](https://mlsanity.com/paper/2104.06770/) (2021)
- [Attention Gate in Traffic Forecasting](https://mlsanity.com/paper/2109.13021/) (2021)
- [Decoupled Object-Centric Video Understanding for Generating Robotic Manipulation Commands](https://mlsanity.com/paper/2606.16470/) (2026)

## Abstract (arXiv)

We propose V2CNet, a new deep learning framework to automatically translate the demonstration videos to commands that can be directly used in robotic applications. Our V2CNet has two branches and aims at understanding the demonstration video in a fine-grained manner. The first branch has the encoder-decoder architecture to encode the visual features and sequentially generate the output words as a command, while the second branch uses a Temporal Convolutional Network (TCN) to learn the fine-grained actions. By jointly training both branches, the network is able to model the sequential information of the command, while effectively encodes the fine-grained actions. The experimental results on our new large-scale dataset show that V2CNet outperforms recent state-of-the-art methods by a substantial margin, while its output can be applied in real robotic applications. The source code and trained models will be made available.