mlsanityData to 7 Oct 2026

Improving 2D Feature Representations by 3D-Aware Fine-Tuning

arXiv 2407.20229 ↗First posted 29 Jul 2024cs.CVcited by 70

Not on the map. Why?

Abstract (arXiv)

Current visual foundation models are trained purely on unstructured 2D data, limiting their understanding of 3D structure of objects and scenes. In this work, we show that fine-tuning on 3D-aware data improves the quality of emerging semantic features. We design a method to lift semantic 2D features into an efficient 3D Gaussian representation, which allows us to re-render them for arbitrary views. Using the rendered 3D-aware features, we design a fine-tuning strategy to transfer such 3D awareness into a 2D foundation model. We demonstrate that models fine-tuned in that way produce features that readily improve downstream task performance in semantic segmentation and depth estimation through simple linear probing. Notably, though fined-tuned on a single indoor dataset, the improvement is transferable to a variety of indoor datasets and out-of-domain datasets. We hope our study encourages the community to consider injecting 3D awareness when training 2D foundation models. Project page: https://ywyue.github.io/FiT3D.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Builds on 56 of its 68 references that are in the map. First posted on arXiv 29 Jul 2024.

Builds on

56 of 68 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2023Segment Anythingcited by 8,158
  8. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  12. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  5. 2021Vision Transformers for Dense Predictioncited by 1,789
  6. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  7. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  8. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  10. 2023Vision Transformers Need Registerscited by 844
  11. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  12. 2022Language-driven Semantic Segmentationcited by 711
  13. 2020AdaBins: Depth Estimation using Adaptive Binscited by 596
  14. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  15. 2023LERF: Language Embedded Radiance Fieldscited by 538
  16. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 471
  17. 2022DeiT III: Revenge of the ViTcited by 417
  18. 2023LangSplat: 3D Language Gaussian Splattingcited by 410
  19. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  20. 2023Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fieldscited by 361
  21. 2022Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentationcited by 341
  22. 2021Deep ViT Features as Dense Visual Descriptorscited by 327
  23. 2023ConceptFusion: Open-set Multimodal 3D Mappingcited by 302
  24. 2014Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networkscited by 299
  25. 2023A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondencecited by 297
  26. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
  27. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  28. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 205
  29. 2022Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representationscited by 195
  30. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  31. 2022CroCo: Self-Supervised Pre-training for 3D Vision Tasks by Cross-View Completioncited by 158
  32. 2016Learning Features by Watching Objects Movecited by 138
  33. 2023Distilled Feature Fields Enable Few-Shot Language-Guided Manipulationcited by 130
  34. 2023The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimationcited by 116
  35. 2024FeatUp: A Model-Agnostic Framework for Features at Any Resolutioncited by 114
  36. 2022Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Modelscited by 95
  37. 2024OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Viewscited by 68
  38. 2023Segment3D: Learning Fine-Grained Class-Agnostic 3D Segmentation without Manual Labelscited by 64

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 39; 2026: 19.

70 papers.

Sort:

By area:

  1. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 63
  2. 2025WildGS-SLAM: Monocular Gaussian Splatting SLAM in Dynamic Environmentscited by 48
  3. 2024DeSiRe-GS: 4D Street Gaussians for Static-Dynamic Decomposition and Surface Reconstruction for Urban Driving Scenescited by 45
  4. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  5. 2024Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generationcited by 42
  6. 2025Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representationscited by 42
  7. 2024LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenescited by 36
  8. 2024SPA: 3D Spatial-Awareness Enables Effective Embodied Representationcited by 28
  9. 2025DUNE: Distilling a Universal Encoder from Heterogeneous 2D and 3D Teacherscited by 28
  10. 2024Feat2GS: Probing Visual Foundation Models with Gaussian Splattingcited by 26
  11. 2024P2P-Bridge: Diffusion Bridges for 3D Point Cloud Denoisingcited by 18
  12. 2024DepthCues: Evaluating Monocular Depth Perception in Large Vision Modelscited by 18
38 more
  1. 2025C3G: Learning Compact 3D Representations with 2K Gaussianscited by 18
  2. 2024LoRA3D: Low-Rank Self-Calibration of 3D Geometric Foundation Modelscited by 17
  3. 2024LangSurf: Language-Embedded Surface Gaussians for 3D Scene Understandingcited by 17
  4. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 14
  5. 2024Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuningcited by 13
  6. 2026INSID3: Training-Free In-Context Segmentation with DINOv3cited by 12
  7. 2025Feed-Forward SceneDINO for Unsupervised Semantic Scene Completioncited by 11
  8. 2025CF3: Compact and Fast 3D Feature Fieldscited by 11
  9. 2024Diorama: Unleashing Zero-shot Single-view 3D Indoor Scene Modelingcited by 10
  10. 2025Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objectscited by 10
  11. 2025Cross-Frame Representation Alignment for Fine-Tuning Video Diffusion Modelscited by 9
  12. 2026DROID-SLAM in the Wildcited by 9
  13. 2025JOG3R: Towards 3D-Consistent Video Generatorscited by 8
  14. 2024ARKit LabelMaker: A New Scale for Indoor 3D Scene Understandingcited by 7
  15. 2025BevSplat: Resolving Height Ambiguity via Feature-Based Gaussian Primitives for Weakly-Supervised Cross-View Localizationcited by 7
  16. 2025Flux4D: Flow-based Unsupervised 4D Reconstructioncited by 7
  17. 2025SAB3R: Semantic-Augmented Backbone in 3D Reconstructioncited by 6
  18. 2025GECO: Geometrically Consistent Embedding with Lightspeed Inferencecited by 6
  19. 2025EgoDTM: Towards 3D-Aware Egocentric Video-Language Pretrainingcited by 5
  20. 2025Are Synthetic Corruptions A Reliable Proxy For Real-World Corruptions?cited by 5
  21. 2025AnyMoLe: Any Character Motion In-betweening Leveraging Video Diffusion Modelscited by 4
  22. 2025SLAG: Scalable Language-Augmented Gaussian Splattingcited by 4
  23. 20253D-Aware Vision-Language Models Fine-Tuning with Geometric Distillationcited by 4
  24. 2025Learning Dense Feature Matching via Lifting Single 2D Image to 3D Spacecited by 4
  25. 2025M3: 3D-Spatial MultiModal Memorycited by 3
  26. 2025View-Consistent Diffusion Representations for 3D-Consistent Video Generationcited by 3
  27. 2025IDSplat: Instance-Decomposed 3D Gaussian Splatting for Driving Scenescited by 3
  28. 2025PanoGrounder: Bridging 2D and 3D with Panoramic Scene Representations for VLM-based 3D Visual Groundingcited by 3
  29. 2026Flow4DGS-SLAM: Optical Flow-Guided 4D Gaussian Splatting SLAMcited by 3
  30. 2025SPC-GS: Gaussian Splatting with Semantic-Prompt Consistency for Indoor Open-World Free-view Synthesis from Sparse Inputscited by 2
  31. 2025Interpretable 3D Neural Object Volumes for Robust Conceptual Reasoningcited by 2
  32. 2025GaussianCross: Cross-modal Self-supervised 3D Representation Learning via Gaussian Splattingcited by 2
  33. 2025Unleashing Semantic and Geometric Priors for 3D Scene Completioncited by 2
  34. 2025Maybe you don't need a U-Net: convolutional feature upsampling for materials micrograph segmentationcited by 2
  35. 2025Multi-View Foundation Modelscited by 2
  36. 2026Revisiting Multi-Task Visual Representation Learningcited by 2
  37. 2026FF3R: Feedforward Feature 3D Reconstruction from Unconstrained viewscited by 2
  38. 2026VEGA: Visual Encoder Grounding Alignment for Spatially-Aware Vision-Language-Action Modelscited by 2

50 most cited of 70 All 70 (JSON)