When Left Is Right? Camera-Motion Understanding beyond Screen
On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs also fits: Embodied and 3D vision-language models
Abstract (arXiv)
Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip. Adding camera-motion data or geometric representations has been tried, separately, and neither closes the gap. We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator. We propose CameraVLM, which pairs a motion-state representation with state-language alignment. Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics. With only 9B parameters, CameraVLM outperforms all open-source and camera-motion-specific models by a wide margin across three benchmarks and more than 20 baselines, and matches or surpasses GPT-6 Astra as well as human viewers in a 48-participant study. Project page: https://3dagentworld.github.io/CameraVLM/
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 64 references that are in the map. First posted on arXiv 12 Oct 2026.
Builds on
47 of 64 references.
Closest ancestors · 4
Same direction, newest first
- 2026Geometry-Guided Camera Motion Understanding in VideoLLMs13 Mar
- 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling21 Jan
- 2024VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling31 Dec
- 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis31 May
Other references · 43
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2024Grounding Image Matching in 3D with MASt3Rcited by 983
- 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
- 2025Continuous 3D Perception Model with Persistent Statecited by 548
- 2024MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motioncited by 442
- 2025Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Passcited by 367
- 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 287
- 2022TAP-Vid: A Benchmark for Tracking Any Point in a Videocited by 263
- 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
- 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 180
- 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
31 more
- 2024CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videoscited by 128
- 2024BootsTAP: Bootstrapped Training for Tracking-Any-Pointcited by 120
- 2025Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memorycited by 87
- 2024Local All-Pair Correspondence for Point Trackingcited by 80
- 2024DELTA: Dense Efficient Long-range 3D Tracking for any videocited by 65
- 2025Think with 3D: Geometric Imagination Grounded Spatial Reasoning from Limited Viewscited by 65
- 2025Towards Understanding Camera Motions in Any Videocited by 64
- 2023Dense Optical Tracking: Connecting the Dotscited by 61
- 2025AllTracker: Efficient Dense Point Tracking at High Resolutioncited by 58
- 2025VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledgecited by 55
- 2024TAPTR: Tracking Any Point with Transformers as Detectioncited by 45
- 2025VGGT4D: Mining Motion Cues in Visual Geometry Transformers for 4D Scene Reconstructioncited by 28
- 2025SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Modelscited by 26
- 2024E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awarenesscited by 25
- 2025CineTechBench: A Benchmark for Cinematographic Technique Understanding and Generationcited by 23
- 2025Outdoor Monocular SLAM with Global Scale-Consistent 3D Gaussian Pointmapscited by 22
- 2025RGB-Only Gaussian Splatting SLAM for Unbounded Outdoor Scenescited by 18
- 2026CoWTracker: Tracking by Warping instead of Correlationcited by 14
- 2025RegGS: Unposed Sparse Views Gaussian Splatting with 3DGS Registrationcited by 13
- 2025Tracktention: Leveraging Point Tracking to Attend Videos Faster and Bettercited by 9
- 2026CamReasoner: Reinforcing Camera Movement Understanding via Structured Spatial Reasoningcited by 9
- 2026HorizonStream: Long-Horizon Attention for Streaming 3D Reconstructioncited by 9
- 2025Graph-Guided Scene Reconstruction from Images with 3D Gaussian Splattingcited by 8
- 2025Unposed 3DGS Reconstruction with Probabilistic Procrustes Mappingcited by 8
- 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agentscited by 8
- 2025LRR-Bench: Left, Right or Rotate? Vision-Language models Still Struggle With Spatial Understanding Taskscited by 6
- 2026Unlocking Dense Metric Depth Estimation in VLMscited by 4
- 2026TrajTok: Learning Trajectory Tokens enables better Video Understandingcited by 3
- 2026Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoningcited by 1
- 2026Natural Language Camera Movement Understandingcited by 1
- 2026Temporally Grounded Compositional Camera Motion Understanding via Geometric Knowledge Distillationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Lost in Space? Vision-Language Models Struggle with Relative Camera Pose Estimation
- 2026Cross-View Action Consistency for Camera-Robust Vision-Language-Action Policies
- 2026Rotated, but How Far? Diagnosing and Improving Object-Rotation Reasoning in VLMs
- 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
- 2026NextMotionQA: Benchmarking and Judging Human Motion Understanding with Vision-Language Models
- 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models
- 2026MotionVLA: Injecting Geometric Motion into Vision-Language-Action Model
- 2026HumanMoveVQA: Can Video MLLMs reason about human movement in videos?