# Towards Understanding Camera Motions in Any Video

arXiv: https://arxiv.org/abs/2504.15376 · HTML: https://mlsanity.com/paper/2504.15376/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 63 of its 88 references that are in the map.
- First posted on arXiv 21 Apr 2025.

## Builds on (most cited first, 30 of 63)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [InternVideo2: Scaling Foundation Models for Multimodal Video Understanding](https://mlsanity.com/paper/2403.15377/) (2024)
- [SkyReels-V2: Infinite-length Film Generative Model](https://mlsanity.com/paper/2504.13074/) (2025)
- [MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos](https://mlsanity.com/paper/2412.04463/) (2024)
- [Visual Geometry Grounded Deep Structure From Motion](https://mlsanity.com/paper/2312.04563/) (2023)
- [MovieNet: A Holistic Dataset for Movie Understanding](https://mlsanity.com/paper/2007.10937/) (2020)
- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [TransNet V2: An effective deep network architecture for fast shot transition detection](https://mlsanity.com/paper/2008.04838/) (2020)
- [Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion](https://mlsanity.com/paper/2402.03162/) (2024)
- [MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion](https://mlsanity.com/paper/2409.19152/) (2024)

## Built on by (most cited first, 30 of 64)

- [ViPE: Video Pose Engine for 3D Geometric Perception](https://mlsanity.com/paper/2508.10934/) (2025)
- [Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding](https://mlsanity.com/paper/2601.10611/) (2026)
- [SpatialVID: A Large-Scale Video Dataset with Spatial Annotations](https://mlsanity.com/paper/2509.09676/) (2025)
- [Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidence](https://mlsanity.com/paper/2510.20579/) (2025)
- [How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspective](https://mlsanity.com/paper/2509.18905/) (2025)
- [OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory](https://mlsanity.com/paper/2512.07802/) (2025)
- [Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings](https://mlsanity.com/paper/2602.13823/) (2026)
- [CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generation](https://mlsanity.com/paper/2505.15145/) (2025)
- [DSI-Bench: A Benchmark for Dynamic Spatial Intelligence](https://mlsanity.com/paper/2510.18873/) (2025)
- [ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models](https://mlsanity.com/paper/2506.21356/) (2025)
- [MonoFusion: Sparse-View 4D Reconstruction via Monocular Fusion](https://mlsanity.com/paper/2507.23782/) (2025)
- [Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation](https://mlsanity.com/paper/2510.08673/) (2025)
- [UltraHR-100K: Enhancing UHR Image Synthesis with A Large-Scale High-Quality Dataset](https://mlsanity.com/paper/2510.20661/) (2025)
- [SpatialTree: How Spatial Abilities Branch Out in MLLMs](https://mlsanity.com/paper/2512.20617/) (2025)
- [RefineShot: Rethinking Cinematography Understanding with Foundational Skill Evaluation](https://mlsanity.com/paper/2510.02423/) (2025)
- [Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum](https://mlsanity.com/paper/2510.27571/) (2025)
- [CamReasoner: Reinforcing Camera Movement Understanding via Structured Spatial Reasoning](https://mlsanity.com/paper/2602.00181/) (2026)
- [Pulp Motion: Framing-aware multimodal camera and human motion generation](https://mlsanity.com/paper/2510.05097/) (2025)
- [TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation](https://mlsanity.com/paper/2510.07249/) (2025)
- [DynamicVerse: A Physically-Aware Multimodal Framework for 4D World Modeling](https://mlsanity.com/paper/2512.03000/) (2025)
- [CETCAM: Camera-Controllable Video Generation via Consistent and Extensible Tokenization](https://mlsanity.com/paper/2512.19020/) (2025)
- [Logics-Parsing-Omni Technical Report](https://mlsanity.com/paper/2603.09677/) (2026)
- [OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer](https://mlsanity.com/paper/2604.24762/) (2026)
- [GS-Agent: Creating 4D Physical Worlds With Generative Simulation](https://mlsanity.com/paper/2607.21522/) (2026)
- [LAMP: Language-Assisted Motion Planning for Controllable Video Generation](https://mlsanity.com/paper/2512.03619/) (2025)
- [VLM-SubtleBench: How Far Are VLMs from Human-Level Subtle Comparative Reasoning?](https://mlsanity.com/paper/2603.07888/) (2026)
- [OpenView: Empowering MLLMs with Out-of-view VQA](https://mlsanity.com/paper/2512.18563/) (2025)
- [Geometry-Guided Camera Motion Understanding in VideoLLMs](https://mlsanity.com/paper/2603.13119/) (2026)
- [HumanScore: Benchmarking Human Motions in Generated Videos](https://mlsanity.com/paper/2604.20157/) (2026)
- [Building a Precise Video Language with Human-AI Oversight](https://mlsanity.com/paper/2604.21718/) (2026)

## Abstract (arXiv)

We introduce CameraBench, a large-scale dataset and benchmark designed to assess and improve camera motion understanding. CameraBench consists of ~3,000 diverse internet videos, annotated by experts through a rigorous multi-stage quality control process. One of our contributions is a taxonomy of camera motion primitives, designed in collaboration with cinematographers. We find, for example, that some motions like "follow" (or tracking) require understanding scene content like moving subjects. We conduct a large-scale human study to quantify human annotation performance, revealing that domain expertise and tutorial-based training can significantly enhance accuracy. For example, a novice may confuse zoom-in (a change of intrinsics) with translating forward (a change of extrinsics), but can be trained to differentiate the two. Using CameraBench, we evaluate Structure-from-Motion (SfM) and Video-Language Models (VLMs), finding that SfM models struggle to capture semantic primitives that depend on scene content, while VLMs struggle to capture geometric primitives that require precise estimation of trajectories. We then fine-tune a generative VLM on CameraBench to achieve the best of both worlds and showcase its applications, including motion-augmented captioning, video question answering, and video-text retrieval. We hope our taxonomy, benchmark, and tutorials will drive future efforts towards the ultimate goal of understanding camera motions in any video.