# Two by Two: Learning Multi-Task Pairwise Objects Assembly for Generalizable Robot Manipulation

arXiv: https://arxiv.org/abs/2504.06961 · HTML: https://mlsanity.com/paper/2504.06961/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 80 references that are in the map.
- First posted on arXiv 9 Apr 2025.

## Builds on (most cited first, 30 of 55)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [SSD-6D: Making RGB-based 3D detection and 6D pose estimation great again](https://mlsanity.com/paper/1711.10006/) (2017)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)
- [PQ-NET: A Generative Part Seq2Seq Network for 3D Shapes](https://mlsanity.com/paper/1911.10949/) (2019)
- [JoinABLe: Learning Bottom-up Assembly of Parametric CAD Joints](https://mlsanity.com/paper/2111.12772/) (2021)
- [UltraLiDAR: Learning Compact Representations for LiDAR Completion and Generation](https://mlsanity.com/paper/2311.01448/) (2023)
- [Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation](https://mlsanity.com/paper/2401.07487/) (2024)
- [Generative 3D Part Assembly via Dynamic Graph Learning](https://mlsanity.com/paper/2006.07793/) (2020)
- [Equivariant Diffusion Policy](https://mlsanity.com/paper/2407.01812/) (2024)
- [Factory: Fast Contact for Robotic Assembly](https://mlsanity.com/paper/2205.03532/) (2022)
- [Learning 3D Part Assembly from a Single Image](https://mlsanity.com/paper/2003.09754/) (2020)
- [Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly](https://mlsanity.com/paper/1910.13675/) (2019)
- [Neural Shape Mating: Self-Supervised Object Assembly with Adversarial Shape Priors](https://mlsanity.com/paper/2205.14886/) (2022)
- [Breaking Bad: A Dataset for Geometric Fracture and Reassembly](https://mlsanity.com/paper/2210.11463/) (2022)
- [Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly](https://mlsanity.com/paper/1903.01066/) (2019)
- [RGL-NET: A Recurrent Graph Learning framework for Progressive Part Assembly](https://mlsanity.com/paper/2107.12859/) (2021)
- [Edge Grasp Network: A Graph-Based SE(3)-invariant Approach to Grasp Detection](https://mlsanity.com/paper/2211.00191/) (2022)

## Built on by (most cited first, 13 of 13)

- [Rectified Point Flow: Generic Point Cloud Pose Estimation](https://mlsanity.com/paper/2506.05282/) (2025)
- [MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning](https://mlsanity.com/paper/2512.16909/) (2025)
- [3D Equivariant Visuomotor Policy Learning via Spherical Projection](https://mlsanity.com/paper/2505.16969/) (2025)
- [BiAssemble: Learning Collaborative Affordance for Bimanual Geometric Assembly](https://mlsanity.com/paper/2506.06221/) (2025)
- [TORA: Topological Representation Alignment for 3D Shape Assembly](https://mlsanity.com/paper/2604.04050/) (2026)
- [Interact3D: Compositional 3D Generation of Interactive Objects](https://mlsanity.com/paper/2603.16085/) (2026)
- [Generalizable Hierarchical Skill Learning via Object-Centric Representation](https://mlsanity.com/paper/2510.21121/) (2025)
- [CADKnitter: Compositional CAD Generation from Text and Geometry Guidance](https://mlsanity.com/paper/2512.11199/) (2025)
- [Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding](https://mlsanity.com/paper/2512.12822/) (2025)
- [Copy-Trasform-Paste: Zero-Shot Object-Object Alignment Guided by Vision-Language and Geometric Constraints](https://mlsanity.com/paper/2601.14207/) (2026)
- [LAMP: Lift Image-Editing as General 3D Priors for Open-world Manipulation](https://mlsanity.com/paper/2604.08475/) (2026)
- [AssemblyWorld: Rethinking 3D Assembly with General-Purpose Agents](https://mlsanity.com/paper/2609.40353/) (2026)
- [PhySGA: A Physics-Space Grounded Agentic Framework for Generalizable and Interactive 3D Composition](https://mlsanity.com/paper/2610.20560/) (2026)

## Abstract (arXiv)

3D assembly tasks, such as furniture assembly and component fitting, play a crucial role in daily life and represent essential capabilities for future home robots. Existing benchmarks and datasets predominantly focus on assembling geometric fragments or factory parts, which fall short in addressing the complexities of everyday object interactions and assemblies. To bridge this gap, we present 2BY2, a large-scale annotated dataset for daily pairwise objects assembly, covering 18 fine-grained tasks that reflect real-life scenarios, such as plugging into sockets, arranging flowers in vases, and inserting bread into toasters. 2BY2 dataset includes 1,034 instances and 517 pairwise objects with pose and symmetry annotations, requiring approaches that align geometric shapes while accounting for functional and spatial relationships between objects. Leveraging the 2BY2 dataset, we propose a two-step SE(3) pose estimation method with equivariant features for assembly constraints. Compared to previous shape assembly methods, our approach achieves state-of-the-art performance across all 18 tasks in the 2BY2 dataset. Additionally, robot experiments further validate the reliability and generalization ability of our method for complex 3D assembly tasks.