mlsanityData to 7 Oct 2026

Self-supervised Learning by View Synthesis

arXiv 2304.11330 ↗First posted 22 Apr 2023cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

We present view-synthesis autoencoders (VSA) in this paper, which is a self-supervised learning framework designed for vision transformers. Different from traditional 2D pretraining methods, VSA can be pre-trained with multi-view data. In each iteration, the input to VSA is one view (or multiple views) of a 3D object and the output is a synthesized image in another target pose. The decoder of VSA has several cross-attention blocks, which use the source view as value, source pose as key, and target pose as query. They achieve cross-attention to synthesize the target view. This simple approach realizes large-angle view synthesis and learns spatial invariant representation, where the latter is decent initialization for transformers on downstream tasks, such as 3D classification on ModelNet40, ShapeNet Core55, and ScanObjectNN. VSA outperforms existing methods significantly for linear probing and is competitive for fine-tuning. The code will be made publicly available.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 57 of its 75 references that are in the map. First posted on arXiv 22 Apr 2023.

Builds on

57 of 75 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2017Neural Discrete Representation Learningcited by 4,650
38 more
  1. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  4. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  5. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  6. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  8. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  9. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  11. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  12. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  13. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  14. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  15. 2020Point Transformercited by 1,462
  16. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  17. 2019Contrastive Multiview Codingcited by 1,297
  18. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  19. 2016A Point Set Generation Network for 3D Object Reconstruction from a Single Imagecited by 972
  20. 2020Pre-Trained Image Processing Transformercited by 891
  21. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  22. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  23. 2019Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervisioncited by 699
  24. 2018Pixel2Mesh: Generating 3D Mesh Models from Single RGB Imagescited by 649
  25. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  26. 2021Transformer in Transformercited by 632
  27. 20163D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstructioncited by 587
  28. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  29. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  30. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  31. 2017Large Batch Training of Convolutional Networkscited by 529
  32. 2017Neural 3D Mesh Renderercited by 437
  33. 2019Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoningcited by 420
  34. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  35. 2019What Do Single-view 3D Reconstruction Networks Learn?cited by 259
  36. 2019Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderercited by 221
  37. 2016Learning a Predictable and Generative Vector Representation for Objectscited by 173
  38. 2020MVTN: Multi-View Transformation Network for 3D Shape Recognitioncited by 140

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 1.

1 paper.

  1. 2023Low-shot Object Learning with Mutual Exclusivity Biascited by 2