mlsanityData to 7 Oct 2026

ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer

arXiv 2204.02389 ↗First posted 5 Apr 2022cs.CV cs.LG cs.ROcited by 73

Not on the map. Why?

Abstract (arXiv)

Objects play a crucial role in our everyday activities. Though multisensory object-centric learning has shown great potential lately, the modeling of objects in prior work is rather unrealistic. ObjectFolder 1.0 is a recent dataset that introduces 100 virtualized objects with visual, acoustic, and tactile sensory data. However, the dataset is small in scale and the multisensory data is of limited quality, hampering generalization to real-world scenarios. We present ObjectFolder 2.0, a large-scale, multisensory dataset of common household objects in the form of implicit neural representations that significantly enhances ObjectFolder 1.0 in three aspects. First, our dataset is 10 times larger in the amount of objects and orders of magnitude faster in rendering time. Second, we significantly improve the multisensory rendering quality for all three modalities. Third, we show that models learned from virtual objects in our dataset successfully transfer to their real-world counterparts in three challenging tasks: object scale estimation, contact localization, and shape reconstruction. ObjectFolder 2.0 offers a new path and testbed for multisensory learning in computer vision and robotics. The dataset is available at https://github.com/rhgao/ObjectFolder.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 48 of its 80 references that are in the map. First posted on arXiv 5 Apr 2022.

Builds on

48 of 80 references.

References · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  5. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  6. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  7. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  8. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
  9. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  10. 2020Neural Sparse Voxel Fieldscited by 959
  11. 2021PlenOctrees for Real-time Rendering of Neural Radiance Fieldscited by 904
  12. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
36 more
  1. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  2. 2021KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPscited by 672
  3. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  4. 2021FastNeRF: High-Fidelity Neural Rendering at 200FPScited by 510
  5. 2021Baking Neural Radiance Fields for Real-Time View Synthesiscited by 465
  6. 2016Deeper Depth Prediction with Fully Convolutional Residual Networkscited by 402
  7. 2018PCN: Point Completion Networkcited by 382
  8. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
  9. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  10. 2018The Sound of Pixelscited by 258
  11. 2021DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networkscited by 232
  12. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  13. 2020AutoInt: Automatic Integration for Fast Neural Volume Renderingcited by 204
  14. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  15. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  16. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  17. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 150
  18. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  19. 2020Music Gesture for Visual Sound Separationcited by 112
  20. 2019Co-Separating Sounds of Visual Objectscited by 110
  21. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  22. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  23. 2020Object-Centric Neural Scene Renderingcited by 90
  24. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  25. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 67
  26. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  27. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 65
  28. 20182.5D Visual Soundcited by 63
  29. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 59
  30. 20203D Shape Reconstruction from Vision and Touchcited by 42
  31. 2021Taxim: An Example-based Simulation Model for GelSight Tactile Sensorscited by 42
  32. 2016The Curious Robot: Learning Visual Representations via Physical Interactionscited by 37
  33. 2017Improved GelSight Tactile Sensor for Measuring Geometry and Slipcited by 27
  34. 2020OmniTact: A Multi-Directional High Resolution Touch Sensorcited by 27
  35. 2020Tactile Object Pose Estimation from the First Touch with Geometric Contact Renderingcited by 19
  36. 2017Localizing the Object Contact through Matching Tactile Features with Visual Mapcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 17; 2024: 15; 2025: 19; 2026: 14.

73 papers.

Sort:

By area:

  1. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  2. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  3. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  4. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  5. 2022Physics-Informed Machine Learning: A Survey on Problems, Methods and Applicationscited by 88
  6. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  7. 2023Rotating without Seeing: Towards In-hand Dexterity through Touchcited by 78
  8. 2023FINER: Flexible spectral-bias tuning in Implicit NEural Representation by Variable-periodic Activation Functionscited by 77
  9. 2024A Touch, Vision, and Language Dataset for Multimodal Alignmentcited by 68
  10. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  11. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 46
  12. 2022Differentiable Physics Simulation of Dynamics-Augmented Neural Objectscited by 45
38 more
  1. 2024Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Taskscited by 44
  2. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  3. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 38
  4. 2025Multimodal Fusion and Vision-Language Models: A Survey for Robot Visioncited by 37
  5. 2024A survey of synthetic data augmentation methods in computer visioncited by 34
  6. 2022DINER: Disorder-Invariant Implicit Neural Representationcited by 30
  7. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 29
  8. 2025Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Grippercited by 29
  9. 2023Disorder-invariant Implicit Neural Representationcited by 26
  10. 2023Generating Visual Scenes from Touchcited by 26
  11. 2024Tactile-Augmented Radiance Fieldscited by 26
  12. 2024Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulationcited by 19
  13. 2023Scene-Driven Multimodal Knowledge Graph Construction for Embodied AIcited by 17
  14. 2023Visual-Tactile Sensing for In-Hand Object Reconstructioncited by 13
  15. 2025OPENTOUCH: Bringing Full-Hand Touch to Real-World Interactioncited by 11
  16. 2026AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perceptioncited by 11
  17. 2023Collision-aware In-hand 6D Object Pose Estimation using Multiple Vision-based Tactile Sensorscited by 10
  18. 2024FINER++: Building a Family of Variable-periodic Functions for Activating Implicit Neural Representationcited by 10
  19. 2023Multi-Object Manipulation via Object-Centric Neural Scattering Functionscited by 9
  20. 2023RHINO: Regularizing the Hash-based Implicit Neural Representationcited by 9
  21. 2025Robotic Compliant Object Prying Using Diffusion Policy Guided by Vision and Force Observationscited by 9
  22. 2023Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hearcited by 8
  23. 2023RealImpact: A Dataset of Impact Sound Fields for Real Objectscited by 8
  24. 2025Universal Visuo-Tactile Video Understanding for Embodied Interactioncited by 7
  25. 2022Neural Shape Compiler: A Unified Framework for Transforming between Text, Point Cloud, and Programcited by 6
  26. 2024Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Datasetcited by 6
  27. 2022Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensingcited by 5
  28. 2023M3Act: Learning from Synthetic Human Group Activitiescited by 5
  29. 2023Aria-NeRF: Multimodal Egocentric View Synthesiscited by 4
  30. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  31. 2024Exploring the Potential of Synthetic Data to Replace Real Datacited by 3
  32. 2025X-Capture: An Open-Source Portable Device for Multi-Sensory Learningcited by 3
  33. 2026DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adaptercited by 3
  34. 2024A Sim2Real Approach for Identifying Task-Relevant Properties in Interpretable Machine Learningcited by 2
  35. 2025RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Datacited by 2
  36. 2025Tactile MNIST: Benchmarking Active Tactile Perceptioncited by 2
  37. 2025Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenescited by 2
  38. 2026TaCo: A Benchmark for Lossless and Lossy Codecs of Heterogeneous Tactile Datacited by 2

50 most cited of 73 All 73 (JSON)