mlsanityData to 14 Oct 2026

The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects

arXiv 2306.00956 ↗First posted 1 Jun 2023cs.CV cs.AI cs.GRcited by 41

Not on the map. Why?

Abstract (arXiv)

We introduce the ObjectFolder Benchmark, a benchmark suite of 10 tasks for multisensory object-centric learning, centered around object recognition, reconstruction, and manipulation with sight, sound, and touch. We also introduce the ObjectFolder Real dataset, including the multisensory measurements for 100 real-world household objects, building upon a newly designed pipeline for collecting the 3D meshes, videos, impact sounds, and tactile readings of real-world objects. We conduct systematic benchmarking on both the 1,000 multisensory neural objects from ObjectFolder, and the real multisensory data from ObjectFolder Real. Our results demonstrate the importance of multisensory perception and reveal the respective roles of vision, audio, and touch for different object-centric learning tasks. By publicly releasing our dataset and benchmark suite, we hope to catalyze and enable new research in multisensory object-centric learning in computer vision, robotics, and beyond. Project page: https://objectfolder.stanford.edu

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 57 of its 86 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

57 of 86 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2017Mask R-CNNcited by 7,277
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  7. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  8. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,041
  9. 2019Contrastive Multiview Codingcited by 1,301
  10. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  11. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  12. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 612
38 more
  1. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  2. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  3. 2017Look, Listen and Learncited by 458
  4. 2018PCN: Point Completion Networkcited by 384
  5. 2022Kubric: A scalable dataset generatorcited by 381
  6. 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 366
  7. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  8. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  9. 2016Deep Visual Foresight for Planning Robot Motioncited by 285
  10. 2018The Sound of Pixelscited by 259
  11. 2021DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networkscited by 232
  12. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 230
  13. 2018Pix3D: Dataset and Methods for Single-Image 3D Shape Modelingcited by 218
  14. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 154
  15. 2021Taming Visually Guided Sound Generationcited by 136
  16. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  17. 2019Co-Separating Sounds of Visual Objectscited by 110
  18. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 102
  19. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  20. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  21. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 87
  22. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  23. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 68
  24. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 68
  25. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 67
  26. 2020Generating Visually Aligned Sound from Videoscited by 67
  27. 2019Connecting Touch and Vision via Cross-Modal Predictioncited by 66
  28. 2020TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensorscited by 63
  29. 2022Visual Acoustic Matchingcited by 45
  30. 2021Taxim: An Example-based Simulation Model for GelSight Tactile Sensorscited by 43
  31. 20203D Shape Reconstruction from Vision and Touchcited by 42
  32. 2021CDPAM: Contrastive learning for perceptual audio similaritycited by 42
  33. 2019Manipulation by Feel: Touch-Based Control with Deep Predictive Modelscited by 36
  34. 2021Active 3D Shape Reconstruction from Vision and Touchcited by 34
  35. 2017See, Hear, and Read: Deep Aligned Representationscited by 32
  36. 2021ShapeMap 3-D: Efficient shape mapping through dense touch and visioncited by 31
  37. 2017Improved GelSight Tactile Sensor for Measuring Geometry and Slipcited by 28
  38. 2022Context is Everything: Implicit Identification for Dynamics Adaptationcited by 27

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 15; 2026: 14.

41 papers.

Sort:

By area:

  1. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 179
  2. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 95
  3. 2025AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensorscited by 48
  4. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 39
  5. 2024ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Datacited by 31
  6. 2023Generating Visual Scenes from Touchcited by 26
  7. 2024Tactile-Augmented Radiance Fieldscited by 26
  8. 2024RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packingcited by 19
  9. 2025OPENTOUCH: Bringing Full-Hand Touch to Real-World Interactioncited by 15
  10. 2024Snap-it, Tap-it, Splat-it: Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfacescited by 12
  11. 2026AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perceptioncited by 12
  12. 2025Robotic Compliant Object Prying Using Diffusion Policy Guided by Vision and Force Observationscited by 10
29 more
  1. 2025ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representationscited by 8
  2. 2024Towards Comprehensive Multimodal Perception: Introducing the Touch-Language-Vision Datasetcited by 6
  3. 2024HyperTaxel: Hyper-Resolution for Taxel-Based Tactile Signals Through Contrastive Learningcited by 6
  4. 2025X-Capture: An Open-Source Portable Device for Multi-Sensory Learningcited by 4
  5. 2025SToLa: Self-Adaptive Touch-Language Framework with Tactile Commonsense Reasoning in Open-Ended Scenarioscited by 3
  6. 2025Tactile MNIST: Benchmarking Active Tactile Perceptioncited by 3
  7. 2025New York Smells: A Large Multimodal Dataset for Olfactioncited by 3
  8. 2023Visually-Guided Sound Source Separation with Audio-Visual Predictive Codingcited by 2
  9. 2024TouchAI: Exploring human-AI perceptual alignment in touch through language model representationscited by 2
  10. 2025ViTaPEs: Visuotactile Position Encodings for Cross-Modal Alignment in Multimodal Transformerscited by 2
  11. 2025Cross-Sensor Touch Generationcited by 2
  12. 2026Seeing Through Touch: Tactile-Driven Visual Localization of Material Regionscited by 2
  13. 2026Tactile-based Multimodal Fusion in Embodied Intelligence: A Survey of Vision, Language, and Contact-Driven Paradigmscited by 2
  14. 2026EgoTac: In-the-wild Tactile Prediction from Egocentric Visioncited by 2
  15. 2025Visual Acoustic Fieldscited by 1
  16. 2025RETRO: REthinking Tactile Representation Learning with Material PriOrscited by 1
  17. 2025VibraVerse: A Large-Scale Geometry-Acoustics Alignment Dataset for Physically-Consistent Multimodal Learningcited by 1
  18. 2026FlowTouch: View-Invariant Visuo-Tactile Predictioncited by 1
  19. 2026SLAT-Phys: Fast Material Property Field Prediction from Structured 3D Latentscited by 1
  20. 2026Tac-DINO: Learning Vision-Tactile Features with Patch Alignmentcited by 1
  21. 2026OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policiescited by 1
  22. 2025Modality Selection and Skill Segmentation via Cross-Modality Attentioncited by 0
  23. 2025ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generationcited by 0
  24. 2026OmniVaT: Single Domain Generalization for Multimodal Visual-Tactile Learningcited by 0
  25. 2026VTLoc: Learning-based Tactile Contact Localization in Visual Point Cloudscited by 0
  26. 2026Objects as Audio-Visual Modal Sound Fieldscited by 0
  27. 2026PhysVGGT: Feed-Forward Dense Physical Property Estimation from A Single Imagecited by 0
  28. 2026OmniFysics-Nano-V2 Technical Report: Understanding the Physical World Across Modalitiescited by 0
  29. 2026PhysicsSliders: Learning Physical Controls for Audio Foley Generationcited by 0