# ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer

arXiv: https://arxiv.org/abs/2204.02389 · HTML: https://mlsanity.com/paper/2204.02389/

## Key facts

- Cited by 75 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 80 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 30 of 48)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields](https://mlsanity.com/paper/2011.12100/) (2020)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [PCN: Point Completion Network](https://mlsanity.com/paper/1808.00671/) (2018)
- [ABO: Dataset and Benchmarks for Real-World 3D Object Understanding](https://mlsanity.com/paper/2110.06199/) (2021)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks](https://mlsanity.com/paper/2103.03231/) (2021)
- [Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling](https://mlsanity.com/paper/1804.04610/) (2018)
- [AutoInt: Automatic Integration for Fast Neural Volume Rendering](https://mlsanity.com/paper/2012.01714/) (2020)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Listen to Look: Action Recognition by Previewing Audio](https://mlsanity.com/paper/1912.04487/) (2019)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)

## Built on by (most cited first, 30 of 75)

- [MVImgNet: A Large-scale Dataset of Multi-view Images](https://mlsanity.com/paper/2303.06042/) (2023)
- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning](https://mlsanity.com/paper/2206.08312/) (2022)
- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [Rotating without Seeing: Towards In-hand Dexterity through Touch](https://mlsanity.com/paper/2303.10880/) (2023)
- [FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functions](https://mlsanity.com/paper/2312.02434/) (2023)
- [A Touch, Vision, and Language Dataset for Multimodal Alignment](https://mlsanity.com/paper/2402.13232/) (2024)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors](https://mlsanity.com/paper/2502.12191/) (2025)
- [Differentiable Physics Simulation of Dynamics-Augmented Neural Objects](https://mlsanity.com/paper/2210.09420/) (2022)
- [Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks](https://mlsanity.com/paper/2406.13640/) (2024)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision](https://mlsanity.com/paper/2504.02477/) (2025)
- [MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World](https://mlsanity.com/paper/2401.08577/) (2024)
- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data](https://mlsanity.com/paper/2406.19464/) (2024)
- [DINER: Disorder-Invariant Implicit Neural Representation](https://mlsanity.com/paper/2211.07871/) (2022)
- [Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper](https://mlsanity.com/paper/2507.15062/) (2025)
- [Disorder-invariant Implicit Neural Representation](https://mlsanity.com/paper/2304.00837/) (2023)
- [Generating Visual Scenes from Touch](https://mlsanity.com/paper/2309.15117/) (2023)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2408.01366/) (2024)
- [Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI](https://mlsanity.com/paper/2311.03783/) (2023)
- [OPENTOUCH: Bringing Full-Hand Touch to Real-World Interaction](https://mlsanity.com/paper/2512.16842/) (2025)
- [Visual-Tactile Sensing for In-Hand Object Reconstruction](https://mlsanity.com/paper/2303.14498/) (2023)
- [AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception](https://mlsanity.com/paper/2602.09617/) (2026)
- [FINER++: Building a Family of Variable-periodic Functions for Activating Implicit Neural Representation](https://mlsanity.com/paper/2407.19434/) (2024)
- [Collision-aware In-hand 6D Object Pose Estimation using Multiple Vision-based Tactile Sensors](https://mlsanity.com/paper/2301.13667/) (2023)

## Abstract (arXiv)

Objects play a crucial role in our everyday activities. Though multisensory object-centric learning has shown great potential lately, the modeling of objects in prior work is rather unrealistic. ObjectFolder 1.0 is a recent dataset that introduces 100 virtualized objects with visual, acoustic, and tactile sensory data. However, the dataset is small in scale and the multisensory data is of limited quality, hampering generalization to real-world scenarios. We present ObjectFolder 2.0, a large-scale, multisensory dataset of common household objects in the form of implicit neural representations that significantly enhances ObjectFolder 1.0 in three aspects. First, our dataset is 10 times larger in the amount of objects and orders of magnitude faster in rendering time. Second, we significantly improve the multisensory rendering quality for all three modalities. Third, we show that models learned from virtual objects in our dataset successfully transfer to their real-world counterparts in three challenging tasks: object scale estimation, contact localization, and shape reconstruction. ObjectFolder 2.0 offers a new path and testbed for multisensory learning in computer vision and robotics. The dataset is available at https://github.com/rhgao/ObjectFolder.