CapFrame: Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes via Geometric Pseudo Labels
Not on the map. Why?
Abstract (arXiv)
3D Gaussian Splatting (3DGS) enables photorealistic real-time novel view synthesis, yet placing a virtual camera to capture a desired frame remains largely manual. Existing language-guided approaches in 3D scenes mainly focus on object-centric grounding, determining what to observe but rarely controlling how it should appear in a single frame, such as subject orientation or frame layout. To address this limitation, we introduce a new task, Text-Instructed Viewpoint Grounding (TIVG), which aims to identify a 6-DoF camera pose in a 3D Gaussian scene whose rendered frame aligns with a text instruction. To solve this task, we propose CapFrame, a partially differentiable framework that converts language into geometric pseudo labels for camera pose optimization. CapFrame follows a Retrieve-Translate-Refine pipeline: it retrieves relevant views and ranks them through a Question-Evaluation process with MLLMs, translates the instruction into orientation and layout pseudo labels, and refines the camera pose via differentiable optimization with layout and orientation losses in 3DGS. Experiments on 38 real-world scenes with 135 instructions indicate that CapFrame produces viewpoints better aligned with texts than heuristic viewpoint search and adapted trajectory generation baselines, validated by VLM metrics, MLLM judges, and user studies. Code is available at: https://github.com/jirongli/CapFrame
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 55 of its 68 references that are in the map. First posted on arXiv 31 Aug 2026.
Builds on
55 of 68 references.
References · 55
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Segment Anythingcited by 8,234
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
- 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
38 more
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,689
- 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
- 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
- 2021Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstructioncited by 964
- 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 929
- 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
- 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
- 2023LERF: Language Embedded Radiance Fieldscited by 543
- 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 537
- 2017Neural 3D Mesh Renderercited by 437
- 2023Gaussian Splatting SLAMcited by 415
- 2023LangSplat: 3D Language Gaussian Splattingcited by 415
- 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 393
- 2023GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splattingcited by 335
- 2018Learning Category-Specific Mesh Reconstruction from Image Collectionscited by 317
- 2023Text-to-3D using Gaussian Splattingcited by 299
- 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
- 2023Language Embedded 3D Gaussians for Open-Vocabulary Scene Understandingcited by 208
- 2024VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Realitycited by 179
- 2024OpenGaussian: Towards Point-Level 3D Gaussian-based Open Vocabulary Understandingcited by 145
- 2022VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Gridscited by 133
- 2024GaussCtrl: Multi-View Consistent Text-Driven 3D Gaussian Splatting Editingcited by 115
- 2024DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splattingcited by 82
- 2025FG-CLIP: Fine-Grained Visual and Textual Alignmentcited by 81
- 2024Click-Gaussian: Interactive Segmentation to Any 3D Gaussianscited by 58
- 2024Director3D: Real-world Camera Trajectory and 3D Scene Generation from Textcited by 50
- 2020End-to-End Learning Local Multi-view Descriptors for 3D Point Cloudscited by 47
- 2024Orient Anything: Learning Robust Object Orientation Estimation from Rendering 3D Modelscited by 45
- 2024SplatLoc: 3D Gaussian Splatting-based Visual Localization for Augmented Realitycited by 43
- 2025LangSplatV2: High-dimensional 3D Language Gaussian Splatting with 450+ FPScited by 41
- 2024Efficient Physics Simulation for 3D Scenes via MLLM-Guided Gaussian Splattingcited by 32
- 2024E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awarenesscited by 25
- 2024GaussianCut: Interactive segmentation via graph cut for 3D Gaussian Splattingcited by 25
- 2025ReasonGrounder: LVLM-Guided Hierarchical Feature Splatting for Open-Vocabulary 3D Visual Grounding and Reasoningcited by 25
- 2025InterGSEdit: Interactive 3D Gaussian Splatting Editing with 3D Geometry-Consistent Attention Priorcited by 19
- 2025GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photographycited by 18
50 most cited of 55 All 55 (JSON)
Built on by
Not cited yet.