# TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation

arXiv: https://arxiv.org/abs/2211.09325 · HTML: https://mlsanity.com/paper/2211.09325/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 64 references that are in the map.
- First posted on arXiv 17 Nov 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Deep Closest Point: Learning Representations for Point Cloud Registration](https://mlsanity.com/paper/1905.03304/) (2019)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation](https://mlsanity.com/paper/2112.05124/) (2021)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [kPAM: KeyPoint Affordances for Category-Level Robotic Manipulation](https://mlsanity.com/paper/1903.06684/) (2019)
- [FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation](https://mlsanity.com/paper/2103.02242/) (2021)
- [Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation](https://mlsanity.com/paper/1806.08756/) (2018)
- [FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects](https://mlsanity.com/paper/2205.04382/) (2022)
- [Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2009.05085/) (2020)
- [KETO: Learning Keypoint Representations for Tool Manipulation](https://mlsanity.com/paper/1910.11977/) (2019)
- [Object Rearrangement Using Learned Implicit Collision Functions](https://mlsanity.com/paper/2011.10726/) (2020)
- [Personalization of End-to-end Speech Recognition On Mobile Devices For Named Entities](https://mlsanity.com/paper/1912.09251/) (2019)
- [Planar Robot Casting with Real2Sim2Real Self-Supervised Learning](https://mlsanity.com/paper/2111.04814/) (2021)
- [Visual Identification of Articulated Object Parts](https://mlsanity.com/paper/2012.00284/) (2020)
- [S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency](https://mlsanity.com/paper/2009.14711/) (2020)
- [Robots of the Lost Arc: Self-Supervised Learning to Dynamically Manipulate Fixed-Endpoint Cables](https://mlsanity.com/paper/2011.04840/) (2020)
- [GIFT: Generalizable Interaction-aware Functional Tool Affordances without Labels](https://mlsanity.com/paper/2106.14973/) (2021)
- [Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms](https://mlsanity.com/paper/2105.14246/) (2021)
- [6-DoF Grasp Planning using Fast 3D Reconstruction and Grasp Quality CNN](https://mlsanity.com/paper/2009.08618/) (2020)
- [ART/ATK: A research platform for assessing and mitigating the sim-to-real gap in robotics and autonomous vehicle engineering](https://mlsanity.com/paper/2211.04886/) (2022)

## Built on by (most cited first, 30 of 47)

- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Flow as the Cross-Domain Manipulation Interface](https://mlsanity.com/paper/2407.15208/) (2024)
- [Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation](https://mlsanity.com/paper/2405.01527/) (2024)
- [General Flow as Foundation Affordance for Scalable Robot Learning](https://mlsanity.com/paper/2401.11439/) (2024)
- [Equivariant Diffusion Policy](https://mlsanity.com/paper/2407.01812/) (2024)
- [Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D](https://mlsanity.com/paper/2401.12046/) (2024)
- [Shelving, Stacking, Hanging: Relational Pose Diffusion for Multi-modal Rearrangement](https://mlsanity.com/paper/2307.04751/) (2023)
- [ActionFlow: Equivariant, Accurate, and Efficient Policies with Spatially Symmetric Flow Matching](https://mlsanity.com/paper/2409.04576/) (2024)
- [DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification](https://mlsanity.com/paper/2305.15957/) (2023)
- [A General Theory of Correct, Incorrect, and Extrinsic Equivariance](https://mlsanity.com/paper/2303.04745/) (2023)
- [Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies](https://mlsanity.com/paper/2406.11740/) (2024)
- [Composable Part-Based Manipulation](https://mlsanity.com/paper/2405.05876/) (2024)
- [One-shot Imitation Learning via Interaction Warping](https://mlsanity.com/paper/2306.12392/) (2023)
- [P3-PO: Prescriptive Point Priors for Visuo-Spatial Generalization of Robot Policies](https://mlsanity.com/paper/2412.06784/) (2024)
- [Few-Shot In-Context Imitation Learning via Implicit Graph Alignment](https://mlsanity.com/paper/2310.12238/) (2023)
- [RiEMann: Near Real-Time SE(3)-Equivariant Robot Manipulation without Point Cloud Segmentation](https://mlsanity.com/paper/2403.19460/) (2024)
- [AnyPlace: Learning Generalized Object Placement for Robot Manipulation](https://mlsanity.com/paper/2502.04531/) (2025)
- [Web2Grasp: Learning Functional Grasps from Web Images of Hand-Object Interactions](https://mlsanity.com/paper/2505.05517/) (2025)
- [APLA: Additional Perturbation for Latent Noise with Adversarial Training Enables Consistency](https://mlsanity.com/paper/2308.12605/) (2023)
- [Tool-as-Interface: Learning Robot Policies from Observing Human Tool Use](https://mlsanity.com/paper/2504.04612/) (2025)
- [MATCH POLICY: A Simple Pipeline from Point Cloud Registration to Manipulation Policies](https://mlsanity.com/paper/2409.15517/) (2024)
- [Constraint-Preserving Data Generation for Visuomotor Policy Learning](https://mlsanity.com/paper/2508.03944/) (2025)
- [Equivariant Offline Reinforcement Learning](https://mlsanity.com/paper/2406.13961/) (2024)
- [CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators](https://mlsanity.com/paper/2407.06141/) (2024)
- [A Practical Guide for Incorporating Symmetry in Diffusion Policy](https://mlsanity.com/paper/2505.13431/) (2025)
- [SE(3)-bi-equivariant Transformers for Point Cloud Assembly](https://mlsanity.com/paper/2407.09167/) (2024)
- [EfficientFlow: Efficient Equivariant Flow Policy Learning for Embodied AI](https://mlsanity.com/paper/2512.02020/) (2025)
- [Slot-Level Robotic Placement via Visual Imitation from Single Human Video](https://mlsanity.com/paper/2504.01959/) (2025)
- [Mask as Supervision: Leveraging Unified Mask Information for Unsupervised 3D Pose Estimation](https://mlsanity.com/paper/2312.07051/) (2023)
- [Learning Correspondence for Deformable Objects](https://mlsanity.com/paper/2405.08996/) (2024)

## Abstract (arXiv)

How do we imbue robots with the ability to efficiently manipulate unseen objects and transfer relevant skills based on demonstrations? End-to-end learning methods often fail to generalize to novel objects or unseen configurations. Instead, we focus on the task-specific pose relationship between relevant parts of interacting objects. We conjecture that this relationship is a generalizable notion of a manipulation task that can transfer to new objects in the same category; examples include the relationship between the pose of a pan relative to an oven or the pose of a mug relative to a mug rack. We call this task-specific pose relationship "cross-pose" and provide a mathematical definition of this concept. We propose a vision-based system that learns to estimate the cross-pose between two objects for a given manipulation task using learned cross-object correspondences. The estimated cross-pose is then used to guide a downstream motion planner to manipulate the objects into the desired pose relationship (placing a pan into the oven or the mug onto the mug rack). We demonstrate our method's capability to generalize to unseen objects, in some cases after training on only 10 demonstrations in the real world. Results show that our system achieves state-of-the-art performance in both simulated and real-world experiments across a number of tasks. Supplementary information and videos can be found at https://sites.google.com/view/tax-pose/home.