# Instruct2Act: From Human Instruction to Actions Sequencing and Execution via Robot Action Network for Robotic Manipulation

arXiv: https://arxiv.org/abs/2602.09940 · HTML: https://mlsanity.com/paper/2602.09940/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 16 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Builds on (most cited first, 13 of 13)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [Physically Grounded Vision-Language Models for Robotic Manipulation](https://mlsanity.com/paper/2309.02561/) (2023)
- [GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstration](https://mlsanity.com/paper/2311.12015/) (2023)
- [Multi-Head Attention: Collaborate Instead of Concatenate](https://mlsanity.com/paper/2006.16362/) (2020)
- [Dynamic Movement Primitives in Robotics: A Tutorial Survey](https://mlsanity.com/paper/2102.03861/) (2021)
- [RobotGPT: Robot Manipulation Learning from ChatGPT](https://mlsanity.com/paper/2312.01421/) (2023)
- [Enhancing the LLM-Based Robot Manipulation Through Human-Robot Collaboration](https://mlsanity.com/paper/2406.14097/) (2024)
- [Understanding Natural Language Instructions for Fetching Daily Objects Using GAN-Based Multimodal Target-Source Classification](https://mlsanity.com/paper/1906.06830/) (2019)
- [LLM-BT: Performing Robotic Adaptive Tasks based on Large Language Models and Behavior Trees](https://mlsanity.com/paper/2404.05134/) (2024)

## Abstract (arXiv)

Robots often struggle to follow free-form human instructions in real-world settings due to computational and sensing limitations. We address this gap with a lightweight, fully on-device pipeline that converts natural-language commands into reliable manipulation. Our approach has two stages: (i) the instruction to actions module (Instruct2Act), a compact BiLSTM with a multi-head-attention autoencoder that parses an instruction into an ordered sequence of atomic actions (e.g., reach, grasp, move, place); and (ii) the robot action network (RAN), which uses the dynamic adaptive trajectory radial network (DATRN) together with a vision-based environment analyzer (YOLOv8) to generate precise control trajectories for each sub-action. The entire system runs on a modest system with no cloud services. On our custom proprietary dataset, Instruct2Act attains 91.5% sub-actions prediction accuracy while retaining a small footprint. Real-robot evaluations across four tasks (pick-place, pick-pour, wipe, and pick-give) yield an overall 90% success; sub-action inference completes in < 3.8s, with end-to-end executions in 30-60s depending on task complexity. These results demonstrate that fine-grained instruction-to-action parsing, coupled with DATRN-based trajectory generation and vision-guided grounding, provides a practical path to deterministic, real-time manipulation in resource-constrained, single-camera settings.