# Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms

arXiv: https://arxiv.org/abs/2105.14246 · HTML: https://mlsanity.com/paper/2105.14246/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 31 references that are in the map.
- First posted on arXiv 29 May 2021.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [Self-supervised 6D Object Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/1909.10159/) (2019)
- [LatentFusion: End-to-End Differentiable Reconstruction and Rendering for Unseen Object Pose Estimation](https://mlsanity.com/paper/1912.00416/) (2019)
- [Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning](https://mlsanity.com/paper/1807.03146/) (2018)
- [6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints](https://mlsanity.com/paper/1910.10750/) (2019)
- [EGAD! an Evolved Grasping Analysis Dataset for diversity and reproducibility in robotic manipulation](https://mlsanity.com/paper/2003.01314/) (2020)
- [Relative Camera Pose Estimation Using Convolutional Neural Networks](https://mlsanity.com/paper/1702.01381/) (2017)
- [Deep Directional Statistics: Pose Estimation with Uncertainty Quantification](https://mlsanity.com/paper/1805.03430/) (2018)
- [Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects](https://mlsanity.com/paper/1906.05105/) (2019)
- [Mechanical Search: Multi-Step Retrieval of a Target Object Occluded by Clutter](https://mlsanity.com/paper/1903.01588/) (2019)
- [A Smooth Representation of Belief over SO(3) for Deep Rotation Learning with Uncertainty](https://mlsanity.com/paper/2006.01031/) (2020)
- [Deep-6DPose: Recovering 6D Object Pose from a Single RGB Image](https://mlsanity.com/paper/1802.10367/) (2018)
- [Robust 6D Object Pose Estimation by Learning RGB-D Features](https://mlsanity.com/paper/2003.00188/) (2020)
- [Pose Estimation for Objects with Rotational Symmetry](https://mlsanity.com/paper/1810.05780/) (2018)

## Built on by (most cited first, 17 of 17)

- [FlowBot3D: Learning 3D Articulation Flow to Manipulate Articulated Objects](https://mlsanity.com/paper/2205.04382/) (2022)
- [TAX-Pose: Task-Specific Cross-Pose Estimation for Robot Manipulation](https://mlsanity.com/paper/2211.09325/) (2022)
- [DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation](https://mlsanity.com/paper/2312.00583/) (2023)
- [DiffCLIP: Leveraging Stable Diffusion for Language Grounded 3D Classification](https://mlsanity.com/paper/2305.15957/) (2023)
- [APLA: Additional Perturbation for Latent Noise with Adversarial Training Enables Consistency](https://mlsanity.com/paper/2308.12605/) (2023)
- [CHAMP: Conformalized 3D Human Multi-Hypothesis Pose Estimators](https://mlsanity.com/paper/2407.06141/) (2024)
- [Kit-Net: Self-Supervised Learning to Kit Novel 3D Objects into Novel 3D Cavities](https://mlsanity.com/paper/2107.05789/) (2021)
- [Learning Correspondence for Deformable Objects](https://mlsanity.com/paper/2405.08996/) (2024)
- [Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions](https://mlsanity.com/paper/2405.16184/) (2024)
- [CUPS: Improving Human Pose-Shape Estimators with Conformalized Deep Uncertainty](https://mlsanity.com/paper/2412.10431/) (2024)
- [Self-supervised Wide Baseline Visual Servoing via 3D Equivariance](https://mlsanity.com/paper/2209.05432/) (2022)
- [H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows](https://mlsanity.com/paper/2510.21769/) (2025)
- [Towards Improving Learning from Demonstration Algorithms via MCMC Methods](https://mlsanity.com/paper/2405.02243/) (2024)
- [Theoretical Analysis for Expectation-Maximization-Based Multi-Model 3D Registration](https://mlsanity.com/paper/2405.08991/) (2024)
- [Rocket Landing Control with Grid Fins and Path-following using MPC](https://mlsanity.com/paper/2405.16191/) (2024)
- [Adapting Skills to Novel Grasps: A Self-Supervised Approach](https://mlsanity.com/paper/2408.00178/) (2024)
- [Self-Supervised Visual Representation Learning: Pretrain-Finetuning or Joint Training?](https://mlsanity.com/paper/2607.13192/) (2026)

## Abstract (arXiv)

Orienting objects is a critical component in the automation of many packing and assembly tasks. We present an algorithm to orient novel objects given a depth image of the object in its current and desired orientation. We formulate a self-supervised objective for this problem and train a deep neural network to estimate the 3D rotation as parameterized by a quaternion, between these current and desired depth images. We then use the trained network in a proportional controller to re-orient objects based on the estimated rotation between the two depth images. Results suggest that in simulation we can rotate unseen objects with unknown geometries by up to 30deg with a median angle error of 1.47deg over 100 random initial/desired orientations each for 22 novel objects. Experiments on physical objects suggest that the controller can achieve a median angle error of 4.2deg over 10 random initial/desired orientations each for 5 objects.