mlsanityData to 14 Oct 2026

CapFrame: Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes via Geometric Pseudo Labels

arXiv 2608.30342 ↗First posted 31 Aug 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

3D Gaussian Splatting (3DGS) enables photorealistic real-time novel view synthesis, yet placing a virtual camera to capture a desired frame remains largely manual. Existing language-guided approaches in 3D scenes mainly focus on object-centric grounding, determining what to observe but rarely controlling how it should appear in a single frame, such as subject orientation or frame layout. To address this limitation, we introduce a new task, Text-Instructed Viewpoint Grounding (TIVG), which aims to identify a 6-DoF camera pose in a 3D Gaussian scene whose rendered frame aligns with a text instruction. To solve this task, we propose CapFrame, a partially differentiable framework that converts language into geometric pseudo labels for camera pose optimization. CapFrame follows a Retrieve-Translate-Refine pipeline: it retrieves relevant views and ranks them through a Question-Evaluation process with MLLMs, translates the instruction into orientation and layout pseudo labels, and refines the camera pose via differentiable optimization with layout and orientation losses in 3DGS. Experiments on 38 real-world scenes with 135 instructions indicate that CapFrame produces viewpoints better aligned with texts than heuristic viewpoint search and adapted trajectory generation baselines, validated by VLM metrics, MLLM judges, and user studies. Code is available at: https://github.com/jirongli/CapFrame

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 55 of its 68 references that are in the map. First posted on arXiv 31 Aug 2026.

Builds on

55 of 68 references.

References · 55

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Segment Anythingcited by 8,234
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  8. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  9. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
38 more
  1. 2025Qwen3-VL Technical Reportcited by 2,793
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
  3. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
  4. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  5. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  6. 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
  7. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
  8. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  9. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  10. 2023LERF: Language Embedded Radiance Fieldscited by 543
  11. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 537
  12. 2017Neural 3D Mesh Renderercited by 437
  13. 2023Gaussian Splatting SLAMcited by 415
  14. 2023LangSplat: 3D Language Gaussian Splattingcited by 415
  15. 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
  16. 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 335
  17. 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
  18. 2023Text-to-3D using Gaussian Splattingcited by 299
  19. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  20. 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
  21. 2024VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Realitycited by 179
  22. 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
  23. 2022VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Gridscited by 133
  24. 2024GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editingcited by 115
  25. 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 82
  26. 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
  27. 2024Click-Gaussian: Interactive Segmentation to Any 3D Gaussianscited by 58
  28. 2024Director3D: Real-world Camera Trajectory and 3D Scene Generation from Textcited by 50
  29. 2020End-to-End Learning Local Multi-view Descriptors for 3D Point Cloudscited by 47
  30. 2024Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Modelscited by 45
  31. 2024SplatLoc: 3D Gaussian Splatting-based Visual Localization for Augmented Realitycited by 43
  32. 2025LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPScited by 41
  33. 2024Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splattingcited by 32
  34. 2024E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awarenesscited by 25
  35. 2024GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splattingcited by 25
  36. 2025ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoningcited by 25
  37. 2025InterGSEdit: Interactive 3D Gaussian Splatting Editing with 3D Geometry-Consistent Attention Priorcited by 19
  38. 2025GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photographycited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Not cited yet.