# Coarse-to-Fine 3D Keyframe Transporter

arXiv: https://arxiv.org/abs/2502.01773 · HTML: https://mlsanity.com/paper/2502.01773/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 3 Feb 2025.

## Placed in

- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation](https://mlsanity.com/paper/2209.05451/) (2022)
- [General E(2)-Equivariant Steerable CNNs](https://mlsanity.com/paper/1911.08251/) (2019)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [RVT: Robotic View Transformer for 3D Object Manipulation](https://mlsanity.com/paper/2306.14896/) (2023)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2306.17817/) (2023)
- [Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching](https://mlsanity.com/paper/1710.01330/) (2017)
- [Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation](https://mlsanity.com/paper/2106.12534/) (2021)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation](https://mlsanity.com/paper/2403.03890/) (2024)
- [Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Study](https://mlsanity.com/paper/1404.3785/) (2014)
- [Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2105.14829/) (2021)
- [SO(2)-Equivariant Reinforcement Learning](https://mlsanity.com/paper/2203.04439/) (2022)
- [Equivariant Diffusion Policy](https://mlsanity.com/paper/2407.01812/) (2024)
- [Equivariant Q Learning in Spatial Action Spaces](https://mlsanity.com/paper/2110.15443/) (2021)
- [Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning](https://mlsanity.com/paper/2206.08321/) (2022)
- [Sample Efficient Grasp Learning Using Equivariant Models](https://mlsanity.com/paper/2202.09468/) (2022)
- [Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation](https://mlsanity.com/paper/2309.02685/) (2023)
- [cuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generation](https://mlsanity.com/paper/2310.17274/) (2023)
- [Equivariant Transporter Network](https://mlsanity.com/paper/2202.09400/) (2022)
- [On-Robot Learning With Equivariant Models](https://mlsanity.com/paper/2203.04923/) (2022)
- [DynPoint: Dynamic Neural Point For View Synthesis](https://mlsanity.com/paper/2310.18999/) (2023)
- [Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detection](https://mlsanity.com/paper/2211.00191/) (2022)
- [Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D](https://mlsanity.com/paper/2401.12046/) (2024)

## Built on by (most cited first, 2 of 2)

- [SE(3)-Equivariant Diffusion Policy in Spherical Fourier Space](https://mlsanity.com/paper/2507.01723/) (2025)
- [Improving Vision-Language-Action Model Fine-Tuning with Structured Stage and Keyframe Supervision](https://mlsanity.com/paper/2606.26801/) (2026)

## Abstract (arXiv)

Recent advances in Keyframe Imitation Learning (IL) have enabled learning-based agents to solve a diverse range of manipulation tasks. However, most approaches ignore the rich symmetries in the problem setting and, as a consequence, are sample-inefficient. This work identifies and utilizes the bi-equivariant symmetry within Keyframe IL to design a policy that generalizes to transformations of both the workspace and the objects grasped by the gripper. We make two main contributions: First, we analyze the bi-equivariance properties of the keyframe action scheme and propose a Keyframe Transporter derived from the Transporter Networks, which evaluates actions using cross-correlation between the features of the grasped object and the features of the scene. Second, we propose a computationally efficient coarse-to-fine SE(3) action evaluation scheme for reasoning the intertwined translation and rotation action. The resulting method outperforms strong Keyframe IL baselines by an average of >10% on a wide range of simulation tasks, and by an average of 55% in 4 physical experiments.