mlsanityData to 14 Oct 2026

3D-MOV: Audio-Visual LSTM Autoencoder for 3D Reconstruction of Multiple Objects from Video

arXiv 2110.02404 ↗First posted 5 Oct 2021cs.CV cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

3D object reconstructions of transparent and concave structured objects, with inferred material properties, remains an open research problem for robot navigation in unstructured environments. In this paper, we propose a multimodal single- and multi-frame neural network for 3D reconstructions using audio-visual inputs. Our trained reconstruction LSTM autoencoder 3D-MOV accepts multiple inputs to account for a variety of surface types and views. Our neural network produces high-quality 3D reconstructions using voxel representation. Based on Intersection-over-Union (IoU), we evaluate against other baseline methods using synthetic audio-visual datasets ShapeNet and Sound20K with impact sounds and bounding box annotations. To the best of our knowledge, our single- and multi-frame model is the first audio-visual reconstruction neural network for 3D geometry and material representation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 68 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

45 of 68 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2016Layer Normalizationcited by 5,669
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  6. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,852
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,130
  9. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  10. 2014Conditional Generative Adversarial Netscited by 2,162
  11. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  12. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
33 more
  1. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 979
  2. 2016Pixel Recurrent Neural Networkscited by 816
  3. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  4. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  5. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 588
  6. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
  7. 2016Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modelingcited by 558
  8. 2017Look, Listen and Learncited by 458
  9. 2016OctNet: Learning Deep 3D Representations at High Resolutionscited by 342
  10. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  11. 2018The Sound of Pixelscited by 259
  12. 2016Learning Temporal Regularity in Video Sequencescited by 254
  13. 2016Volumetric and Multi-View CNNs for Object Classification on 3D Datacited by 240
  14. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  15. 2016Learning a Predictable and Generative Vector Representation for Objectscited by 173
  16. 2018Point Convolutional Neural Networks by Extension Operatorscited by 154
  17. 2017Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answeringcited by 117
  18. 2016Learning What and Where to Drawcited by 105
  19. 2017ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scanscited by 103
  20. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  21. 2016Learning shape correspondence with anisotropic convolutional neural networkscited by 95
  22. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 84
  23. 2019MUREL: Multimodal Relational Reasoning for Visual Question Answeringcited by 68
  24. 2019Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Predictioncited by 66
  25. 2017Abnormal Event Detection in Videos using Spatiotemporal Autoencodercited by 57
  26. 2018Im2Struct: Recovering 3D Shape Structure from a Single RGB Imagecited by 55
  27. 2016Discriminative Correlation Filter with Channel and Spatial Reliabilitycited by 45
  28. 20173D Shape Reconstruction from Sketches via Multi-view Convolutional Networkscited by 45
  29. 2018tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flowcited by 44
  30. 2020Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual Recognitioncited by 25
  31. 2020Predictive and Generative Neural Networks for Object Functionalitycited by 9
  32. 2017AlignGAN: Learning to Align Cross-Domain Images with Conditional Generative Adversarial Networkscited by 7
  33. 2018Customizing an Adversarial Example Generator with Class-Conditional GANscited by 1

Built on by

Not cited yet.