# Bi-HIL: Bilateral Control-Based Multimodal Hierarchical Imitation Learning via Subtask-Level Progress Rate and Keyframe Memory for Long-Horizon Contact-Rich Robotic Manipulation

arXiv: https://arxiv.org/abs/2603.13315 · HTML: https://mlsanity.com/paper/2603.13315/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation](https://mlsanity.com/paper/2401.02117/) (2024)
- [GELLO: A General, Low-Cost, and Intuitive Teleoperation Framework for Robot Manipulators](https://mlsanity.com/paper/2309.13037/) (2023)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation](https://mlsanity.com/paper/2502.05485/) (2025)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [Vision Language Models are In-Context Value Learners](https://mlsanity.com/paper/2411.04549/) (2024)
- [FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning](https://mlsanity.com/paper/2401.08553/) (2024)
- [MemER: Scaling Up Memory for Robot Control via Experience Retrieval](https://mlsanity.com/paper/2510.20328/) (2025)
- [ALOHA 2: An Enhanced Low-Cost Hardware for Bimanual Teleoperation](https://mlsanity.com/paper/2405.02292/) (2024)
- [SARM: Stage-Aware Reward Modeling for Long Horizon Robot Manipulation](https://mlsanity.com/paper/2509.25358/) (2025)
- [TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics](https://mlsanity.com/paper/2602.19313/) (2026)
- [Bi-ACT: Bilateral Control-Based Imitation Learning via Action Chunking with Transformer](https://mlsanity.com/paper/2401.17698/) (2024)
- [Imitation Learning for Object Manipulation Based on Position/Force Information Using Bilateral Control](https://mlsanity.com/paper/1811.03759/) (2018)
- [BPP: Long-Context Robot Imitation Learning by Focusing on Key History Frames](https://mlsanity.com/paper/2602.15010/) (2026)
- [Imitation Learning for Variable Speed Contact Motion for Operation up to Control Bandwidth](https://mlsanity.com/paper/2102.10283/) (2021)
- [An Independently Learnable Hierarchical Model for Bilateral Control-Based Imitation Learning Applications](https://mlsanity.com/paper/2203.08619/) (2022)
- [A Survey on Imitation Learning for Contact-Rich Tasks in Robotics](https://mlsanity.com/paper/2506.13498/) (2025)
- [PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement](https://mlsanity.com/paper/2411.17764/) (2024)
- [Time Series Motion Generation Considering Long Short-Term Motion](https://mlsanity.com/paper/1909.03755/) (2019)
- [ILBiT: Imitation Learning for Robot Using Position and Torque Information based on Bilateral Control with Transformer](https://mlsanity.com/paper/2401.16653/) (2024)
- [DABI: Evaluation of Data Augmentation Methods Using Downsampling in Bilateral Control-Based Imitation Learning with Images](https://mlsanity.com/paper/2410.04370/) (2024)
- [Bi-LAT: Bilateral Control-Based Imitation Learning via Natural Language and Action Chunking with Transformers](https://mlsanity.com/paper/2504.01301/) (2025)
- [Bi-VLA: Bilateral Control-Based Imitation Learning via Vision-Language Fusion for Action Generation](https://mlsanity.com/paper/2509.18865/) (2025)

## Built on by (most cited first, 2 of 2)

- [TRACE: Trajectory-Routed Causal Memory for Delayed-Evidence Visuomotor Imitation](https://mlsanity.com/paper/2606.14551/) (2026)
- [PACE: Phase-Progress-Aware Credit for Long-Horizon Embodied Manipulation](https://mlsanity.com/paper/2608.15026/) (2026)

## Abstract (arXiv)

Long-horizon contact-rich robotic manipulation remains challenging due to partial observability and unstable subtask transitions under contact uncertainty. While hierarchical architectures improve temporal reasoning and bilateral imitation learning enables force-aware control, existing approaches often rely on flat policies that struggle with long-horizon coordination. We propose Bi-HIL, a bilateral control-based multimodal hierarchical imitation learning framework for long-horizon manipulation. Bi-HIL stabilizes hierarchical coordination by integrating keyframe memory with subtask-level progress rate that models phase progression within the active subtask and conditions both high- and low-level policies. We evaluate Bi-HIL on unimanual and bimanual real-robot tasks, demonstrating consistent improvements over flat and ablated variants. The results highlight the importance of explicitly modeling subtask progression together with force-aware control for robust long-horizon manipulation. For additional material, please check: https://mertcookimg.github.io/bi-hil