# TO-Scene: A Large-scale Dataset for Understanding 3D Tabletop Scenes

arXiv: https://arxiv.org/abs/2203.09440 · HTML: https://mlsanity.com/paper/2203.09440/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 63 references that are in the map.
- First posted on arXiv 17 Mar 2022.

## Builds on (most cited first, 30 of 40)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [KPConv: Flexible and Deformable Convolution for Point Clouds](https://mlsanity.com/paper/1904.08889/) (2019)
- [3D Semantic Segmentation with Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1711.10275/) (2017)
- [PointConv: Deep Convolutional Networks on 3D Point Clouds](https://mlsanity.com/paper/1811.07246/) (2018)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Deep Hough Voting for 3D Object Detection in Point Clouds](https://mlsanity.com/paper/1904.09664/) (2019)
- [Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data](https://mlsanity.com/paper/1908.04616/) (2019)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [3D-FRONT: 3D Furnished Rooms with layOuts and semaNTics](https://mlsanity.com/paper/2011.09127/) (2020)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [ABC: A Big CAD Model Dataset For Geometric Deep Learning](https://mlsanity.com/paper/1812.06216/) (2018)
- [3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions](https://mlsanity.com/paper/1603.08182/) (2016)
- [An End-to-End Transformer Model for 3D Object Detection](https://mlsanity.com/paper/2109.08141/) (2021)
- [Group-Free 3D Object Detection via Transformers](https://mlsanity.com/paper/2104.00678/) (2021)
- [PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds](https://mlsanity.com/paper/2103.14635/) (2021)
- [Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations](https://mlsanity.com/paper/2012.09988/) (2020)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [H3DNet: 3D Object Detection Using Hybrid Geometric Primitives](https://mlsanity.com/paper/2006.05682/) (2020)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection](https://mlsanity.com/paper/2112.00322/) (2021)
- [Bidirectional Projection Network for Cross Dimension Scene Understanding](https://mlsanity.com/paper/2103.14326/) (2021)
- [Jacquard: A Large Scale Dataset for Robotic Grasp Detection](https://mlsanity.com/paper/1803.11469/) (2018)

## Built on by (most cited first, 10 of 10)

- [MVImgNet: A Large-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2303.06042/) (2023)
- [DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning](https://mlsanity.com/paper/2503.15265/) (2025)
- [Synthetic Datasets for Autonomous Driving: A Survey](https://mlsanity.com/paper/2304.12205/) (2023)
- [SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2311.17707/) (2023)
- [EFEM: Equivariant Neural Field Expectation Maximization for 3D Object Segmentation Without Scene Supervision](https://mlsanity.com/paper/2303.15440/) (2023)
- [MM-3DScene: 3D Scene Understanding by Customizing Masked Modeling with Informative-Preserved Reconstruction and Self-Distilled Consistency](https://mlsanity.com/paper/2212.09948/) (2022)
- [3D Small Object Detection with Dynamic Spatial Pruning](https://mlsanity.com/paper/2305.03716/) (2023)
- [Stable-Sim2Real: Exploring Simulation of Real-Captured 3D Data with Two-Stage Depth Diffusion](https://mlsanity.com/paper/2507.23483/) (2025)
- [GOPLA: Generalizable Object Placement Learning via Synthetic Augmentation of Human Arrangement](https://mlsanity.com/paper/2510.14627/) (2025)
- [3D Adaptive Structural Convolution Network for Domain-Invariant Point Cloud Recognition](https://mlsanity.com/paper/2407.04833/) (2024)

## Abstract (arXiv)

Many basic indoor activities such as eating or writing are always conducted upon different tabletops (e.g., coffee tables, writing desks). It is indispensable to understanding tabletop scenes in 3D indoor scene parsing applications. Unfortunately, it is hard to meet this demand by directly deploying data-driven algorithms, since 3D tabletop scenes are rarely available in current datasets. To remedy this defect, we introduce TO-Scene, a large-scale dataset focusing on tabletop scenes, which contains 20,740 scenes with three variants. To acquire the data, we design an efficient and scalable framework, where a crowdsourcing UI is developed to transfer CAD objects from ModelNet and ShapeNet onto tables from ScanNet, then the output tabletop scenes are simulated into real scans and annotated automatically. Further, a tabletop-aware learning strategy is proposed for better perceiving the small-sized tabletop instances. Notably, we also provide a real scanned test set TO-Real to verify the practical value of TO-Scene. Experiments show that the algorithms trained on TO-Scene indeed work on the realistic test data, and our proposed tabletop-aware learning strategy greatly improves the state-of-the-art results on both 3D semantic segmentation and object detection tasks. Dataset and code are available at https://github.com/GAP-LAB-CUHK-SZ/TO-Scene.