# Bi-ACT: Bilateral Control-Based Imitation Learning via Action Chunking with Transformer

arXiv: https://arxiv.org/abs/2401.17698 · HTML: https://mlsanity.com/paper/2401.17698/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 32 references that are in the map.
- First posted on arXiv 31 Jan 2024.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators](https://mlsanity.com/paper/2309.13037/) (2023)
- [A survey of robot learning from demonstrations for Human-Robot Collaboration](https://mlsanity.com/paper/1710.08789/) (2017)
- [Grasping with Chopsticks: Combating Covariate Shift in Model-free Imitation Learning for Fine Manipulation](https://mlsanity.com/paper/2011.06719/) (2020)
- [Learning Multi-Arm Manipulation Through Collaborative Teleoperation](https://mlsanity.com/paper/2012.06738/) (2020)
- [Causal Imitation Learning under Temporally Correlated Noise](https://mlsanity.com/paper/2202.01312/) (2022)
- [Imitation Learning for Object Manipulation Based on Position/Force Information Using Bilateral Control](https://mlsanity.com/paper/1811.03759/) (2018)
- [Interactive Imitation Learning of Bimanual Movement Primitives](https://mlsanity.com/paper/2210.16220/) (2022)
- [One ACT Play: Single Demonstration Behavior Cloning with Action Chunking Transformers](https://mlsanity.com/paper/2309.10175/) (2023)
- [Imitation Learning for Variable Speed Contact Motion for Operation up to Control Bandwidth](https://mlsanity.com/paper/2102.10283/) (2021)
- [Soft and Rigid Object Grasping With Cross-Structure Hand Using Bilateral Control-Based Imitation Learning](https://mlsanity.com/paper/2311.09555/) (2023)
- [An Independently Learnable Hierarchical Model for Bilateral Control-Based Imitation Learning Applications](https://mlsanity.com/paper/2203.08619/) (2022)
- [Imitation Learning for Nonprehensile Manipulation through Self-Supervised Learning Considering Motion Speed](https://mlsanity.com/paper/2206.10289/) (2022)
- [Motion Generation Using Bilateral Control-Based Imitation Learning with Autoregressive Learning](https://mlsanity.com/paper/2011.06192/) (2020)
- [Time Series Motion Generation Considering Long Short-Term Motion](https://mlsanity.com/paper/1909.03755/) (2019)
- [ILBiT: Imitation Learning for Robot Using Position and Torque Information based on Bilateral Control with Transformer](https://mlsanity.com/paper/2401.16653/) (2024)

## Built on by (most cited first, 27 of 27)

- [Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulation](https://mlsanity.com/paper/2503.02881/) (2025)
- [Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications](https://mlsanity.com/paper/2510.07077/) (2025)
- [RoboCopilot: Human-in-the-loop Interactive Imitation Learning for Robot Manipulation](https://mlsanity.com/paper/2503.07771/) (2025)
- [Learning Variable Compliance Control From a Few Demonstrations for Bimanual Robot with Haptic Feedback Teleoperation System](https://mlsanity.com/paper/2406.14990/) (2024)
- [Learning Bimanual Manipulation via Action Chunking and Inter-Arm Coordination with Transformers](https://mlsanity.com/paper/2503.13916/) (2025)
- [DABI: Evaluation of Data Augmentation Methods Using Downsampling in Bilateral Control-Based Imitation Learning with Images](https://mlsanity.com/paper/2410.04370/) (2024)
- [Bi-LAT: Bilateral Control-Based Imitation Learning via Natural Language and Action Chunking with Transformers](https://mlsanity.com/paper/2504.01301/) (2025)
- [YUBI: Yielding Universal Bidigital Interface for Bimanual Dexterous Manipulation at Scale](https://mlsanity.com/paper/2606.10244/) (2026)
- [Bi-VLA: Bilateral Control-Based Imitation Learning via Vision-Language Fusion for Action Generation](https://mlsanity.com/paper/2509.18865/) (2025)
- [MA-VLA: Multi-Arm Vision-Language-Action Model for Collaboration and Compositional Generalization](https://mlsanity.com/paper/2608.25864/) (2026)
- [Variable-Speed Teaching-Playback as Real-World Data Augmentation for Imitation Learning](https://mlsanity.com/paper/2412.03252/) (2024)
- [Design and Experimental Validation of Sensorless 4-Channel Bilateral Teleoperation for Low-Cost Manipulators](https://mlsanity.com/paper/2507.06174/) (2025)
- [GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies](https://mlsanity.com/paper/2511.00998/) (2025)
- [Bi-HIL: Bilateral Control-Based Multimodal Hierarchical Imitation Learning via Subtask-Level Progress Rate and Keyframe Memory for Long-Horizon Contact-Rich Robotic Manipulation](https://mlsanity.com/paper/2603.13315/) (2026)
- [FACTR 2: Learning External Force Sensing for Commodity Robot Arms Improves Policy Learning](https://mlsanity.com/paper/2606.12406/) (2026)
- [Error-Feedback Model for Output Correction in Bilateral Control-Based Imitation Learning](https://mlsanity.com/paper/2411.12255/) (2024)
- [Improving Low-Cost Teleoperation: Augmenting GELLO with Force](https://mlsanity.com/paper/2507.13602/) (2025)
- [RoboGene: Boosting VLA Pre-training via Diversity-Driven Agentic Framework for Real-World Task Generation](https://mlsanity.com/paper/2602.16444/) (2026)
- [Bimanual Robot Manipulation via Multi-Agent In-Context Learning](https://mlsanity.com/paper/2604.20348/) (2026)
- [FWBC-VLA: Force-Aware Whole-Body Compensation for Contact-Rich Loco-Manipulation](https://mlsanity.com/paper/2609.03889/) (2026)
- [BEAC: Imitating Complex Exploration and Task-oriented Behaviors for Invisible Object Nonprehensile Manipulation](https://mlsanity.com/paper/2503.16803/) (2025)
- [A3RNN: Bi-directional Fusion of Bottom-up and Top-down Process for Developmental Visual Attention in Robots](https://mlsanity.com/paper/2510.10221/) (2025)
- [VRScout: Towards Real-Time, Autonomous Testing of Virtual Reality Games](https://mlsanity.com/paper/2511.00002/) (2025)
- [Force Generative Imitation Learning: Bridging Position Trajectory and Force Commands through Control Technique](https://mlsanity.com/paper/2602.06620/) (2026)
- [Active Stereo-Camera Outperforms Multi-Sensor Setup in ACT Imitation Learning for Humanoid Manipulation](https://mlsanity.com/paper/2603.28422/) (2026)
- [Continuous Manifold-Decomposed Impedance Retargeting for Contact-Rich Imitation Learning](https://mlsanity.com/paper/2609.15716/) (2026)
- [Bi-MoDe: Bilateral Control-based Imitation Learning via Modifier-Conditioned Decoding for Modulation of Execution Speed and Contact Intensity](https://mlsanity.com/paper/2609.16040/) (2026)

## Abstract (arXiv)

Autonomous manipulation in robot arms is a complex and evolving field of study in robotics. This paper proposes work stands at the intersection of two innovative approaches in the field of robotics and machine learning. Inspired by the Action Chunking with Transformer (ACT) model, which employs joint location and image data to predict future movements, our work integrates principles of Bilateral Control-Based Imitation Learning to enhance robotic control. Our objective is to synergize these techniques, thereby creating a more robust and efficient control mechanism. In our approach, the data collected from the environment are images from the gripper and overhead cameras, along with the joint angles, angular velocities, and forces of the follower robot using bilateral control. The model is designed to predict the subsequent steps for the joint angles, angular velocities, and forces of the leader robot. This predictive capability is crucial for implementing effective bilateral control in the follower robot, allowing for more nuanced and responsive maneuvering.