# ChatCam: Empowering Camera Control through Conversational AI

arXiv: https://arxiv.org/abs/2409.17331 · HTML: https://mlsanity.com/paper/2409.17331/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 90 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 30 of 80)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)

## Built on by (most cited first, 15 of 15)

- [Towards Understanding Camera Motions in Any Video](https://mlsanity.com/paper/2504.15376/) (2025)
- [Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generation](https://mlsanity.com/paper/2412.21117/) (2024)
- [WorldCraft: Photo-Realistic 3D World Creation and Customization via LLM Agents](https://mlsanity.com/paper/2502.15601/) (2025)
- [MotionAgent: Fine-grained Controllable Video Generation via Motion Field Agent](https://mlsanity.com/paper/2502.03207/) (2025)
- [LensCraft: Your Professional Virtual Cinematographer](https://mlsanity.com/paper/2506.00988/) (2025)
- [Agentic Aerial Cinematography: From Dialogue Cues to Cinematic Trajectories](https://mlsanity.com/paper/2509.16176/) (2025)
- [Mind-of-Director: Multi-modal Agent-Driven Film Previsualization via Collaborative Decision-Making](https://mlsanity.com/paper/2603.14790/) (2026)
- [CT-1: Vision-Language-Camera Models Transfer Spatial Reasoning Knowledge to Camera-Controllable Video Generation](https://mlsanity.com/paper/2604.09201/) (2026)
- [CinemaTraj: Composing Atomic Camera Trajectories for 3D Scenes with LLM Agents](https://mlsanity.com/paper/2607.26910/) (2026)
- [StateFlow: Building, Evolving, and Accessing 3D World States for Previsualization](https://mlsanity.com/paper/2608.12314/) (2026)
- [VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos](https://mlsanity.com/paper/2510.24904/) (2025)
- [Sensorium Arc: AI Agent System for Oceanic Data Exploration and Interactive Eco-Art](https://mlsanity.com/paper/2511.15997/) (2025)
- [PhotoFlow: Agentic 3D Virtual Photography Missions](https://mlsanity.com/paper/2605.23771/) (2026)
- [Before the Shutter: Aesthetic and Actionable Portrait Photography Planning in 3D Scenes](https://mlsanity.com/paper/2605.30318/) (2026)
- [CapFrame: Text-Instructed Viewpoint Grounding in 3D Gaussian Scenes via Geometric Pseudo Labels](https://mlsanity.com/paper/2608.30342/) (2026)

## Abstract (arXiv)

Cinematographers adeptly capture the essence of the world, crafting compelling visual narratives through intricate camera movements. Witnessing the strides made by large language models in perceiving and interacting with the 3D world, this study explores their capability to control cameras with human language guidance. We introduce ChatCam, a system that navigates camera movements through conversations with users, mimicking a professional cinematographer's workflow. To achieve this, we propose CineGPT, a GPT-based autoregressive model for text-conditioned camera trajectory generation. We also develop an Anchor Determinator to ensure precise camera trajectory placement. ChatCam understands user requests and employs our proposed tools to generate trajectories, which can be used to render high-quality video footage on radiance field representations. Our experiments, including comparisons to state-of-the-art approaches and user studies, demonstrate our approach's ability to interpret and execute complex instructions for camera operation, showing promising applications in real-world production settings.