# InterACT: Inter-dependency Aware Action Chunking with Hierarchical Attention Transformers for Bimanual Manipulation

arXiv: https://arxiv.org/abs/2409.07914 · HTML: https://mlsanity.com/paper/2409.07914/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Behavior Transformers: Cloning k modes with one stone](https://mlsanity.com/paper/2206.11251/) (2022)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [One-Shot Visual Imitation Learning via Meta-Learning](https://mlsanity.com/paper/1709.04905/) (2017)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [The Surprising Effectiveness of Representation Learning for Visual Imitation](https://mlsanity.com/paper/2112.01511/) (2021)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learning](https://mlsanity.com/paper/2206.08686/) (2022)
- [ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperation](https://mlsanity.com/paper/2405.02292/) (2024)
- [Waypoint-Based Imitation Learning for Robotic Manipulation](https://mlsanity.com/paper/2307.14326/) (2023)
- [Multi-Stage Cable Routing through Hierarchical Imitation Learning](https://mlsanity.com/paper/2307.08927/) (2023)
- [Efficient Bimanual Manipulation Using Learned Task Schemas](https://mlsanity.com/paper/1909.13874/) (2019)
- [Stabilize to Act: Learning to Coordinate for Bimanual Manipulation](https://mlsanity.com/paper/2309.01087/) (2023)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [Hi-Transformer: Hierarchical Interactive Transformer for Efficient and Effective Long Document Modeling](https://mlsanity.com/paper/2106.01040/) (2021)
- [Deep Reinforcement Learning for Vision-Based Robotic Grasping: A Simulated Comparative Evaluation of Off-Policy Methods](https://mlsanity.com/paper/1802.10264/) (2018)
- [DenseMTL: Cross-task Attention Mechanism for Dense Multi-task Learning](https://mlsanity.com/paper/2206.08927/) (2022)
- [Hyperdecoders: Instance-specific decoders for multi-task NLP](https://mlsanity.com/paper/2203.08304/) (2022)
- [Multimodal Affective Analysis Using Hierarchical Attention Strategy with Word-Level Alignment](https://mlsanity.com/paper/1805.08660/) (2018)
- [Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularity](https://mlsanity.com/paper/2404.03570/) (2024)
- [ScrewMimic: Bimanual Imitation from Human Videos with Screw Space Projection](https://mlsanity.com/paper/2405.03666/) (2024)
- [Bi-Manual Manipulation and Attachment via Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2203.08277/) (2022)

## Built on by (most cited first, 22 of 22)

- [VITA: Vision-to-Action Flow Matching Policy](https://mlsanity.com/paper/2507.13231/) (2025)
- [Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Reward](https://mlsanity.com/paper/2508.11143/) (2025)
- [TwinVLA: Data-Efficient Bimanual Manipulation with Twin Single-Arm Vision-Language-Action Models](https://mlsanity.com/paper/2511.05275/) (2025)
- [Rethinking Bimanual Robotic Manipulation: Learning with Decoupled Interaction Framework](https://mlsanity.com/paper/2503.09186/) (2025)
- [Look, Focus, Act: Efficient and Robust Robot Learning via Human Gaze and Foveated Vision Transformers](https://mlsanity.com/paper/2507.15833/) (2025)
- [Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers](https://mlsanity.com/paper/2503.13916/) (2025)
- [IN-RIL: Interleaved Reinforcement and Imitation Learning for Policy Fine-Tuning](https://mlsanity.com/paper/2505.10442/) (2025)
- [Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation](https://mlsanity.com/paper/2602.23814/) (2026)
- [Bootstrapping Imitation Learning for Long-horizon Manipulation via Hierarchical Data Collection Space](https://mlsanity.com/paper/2505.17389/) (2025)
- [MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization](https://mlsanity.com/paper/2608.25864/) (2026)
- [Towards a Generalizable Bimanual Foundation Policy via Flow-based Video Prediction](https://mlsanity.com/paper/2505.24156/) (2025)
- [MarineFormer: A Spatio-Temporal Attention Model for USV Navigation in Dynamic Marine Environments](https://mlsanity.com/paper/2410.13973/) (2024)
- [CUBic: Coordinated Unified Bimanual Perception and Control Framework](https://mlsanity.com/paper/2605.13452/) (2026)
- [See Selectively, Act Adaptively: Dual-Level Structural Decomposition for Bimanual Robot Manipulation](https://mlsanity.com/paper/2606.13279/) (2026)
- [Wavelet Policy: Imitation Learning in the Scale Domain with World Prior Memory](https://mlsanity.com/paper/2504.04991/) (2025)
- [RAMAC: Multimodal Risk-Aware Offline Reinforcement Learning and the Role of Behavior Regularization](https://mlsanity.com/paper/2510.02695/) (2025)
- [A Coordinated Dual-Arm Framework for Delicate Snap-Fit Assemblies](https://mlsanity.com/paper/2511.18153/) (2025)
- [SEVO: Semantic-Enhanced Virtual Observation for Robust VLA Manipulation via Active Illumination and Data-Centric Collection](https://mlsanity.com/paper/2605.11114/) (2026)
- [One Hand Watches The Other: Dynamic Multi-Agent Cooperation for Sample-Efficient Bimanual Manipulation in Dynamic Environments](https://mlsanity.com/paper/2607.22119/) (2026)
- [Bimanual Manipulation Within an 8 GB Budget: Zero-Copy Sensing and Quantized ACT on an Entry-Level Jetson](https://mlsanity.com/paper/2608.03938/) (2026)
- [GeoAAC: Geometry-Based Adaptive Action Chunking from Denoising Trajectories in VLA Policies](https://mlsanity.com/paper/2609.20776/) (2026)
- [CoLA: Multi-VLA Coordination via Learned Adapters](https://mlsanity.com/paper/2610.15337/) (2026)

## Abstract (arXiv)

Bimanual manipulation presents unique challenges compared to unimanual tasks due to the complexity of coordinating two robotic arms. In this paper, we introduce InterACT: Inter-dependency aware Action Chunking with Hierarchical Attention Transformers, a novel imitation learning framework designed specifically for bimanual manipulation. InterACT leverages hierarchical attention mechanisms to effectively capture inter-dependencies between dual-arm joint states and visual inputs. The framework comprises a Hierarchical Attention Encoder, which processes multi-modal inputs through segment-wise and cross-segment attention mechanisms, and a Multi-arm Decoder that generates each arm's action predictions in parallel, while sharing information between the arms through synchronization blocks by providing the other arm's intermediate output as context. Our experiments, conducted on various simulated and real-world bimanual manipulation tasks, demonstrate that InterACT outperforms existing methods. Detailed ablation studies further validate the significance of key components, including the impact of CLS tokens, cross-segment encoders, and synchronization blocks on task performance. We provide supplementary materials and videos on our project page.