# TKCAM: Text and Keyframe to Camera Trajectory Generation

arXiv: https://arxiv.org/abs/2610.11105 · HTML: https://mlsanity.com/paper/2610.11105/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 39 references that are in the map.
- First posted on arXiv 8 Oct 2026.

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis](https://mlsanity.com/paper/2409.02048/) (2024)
- [GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium](https://mlsanity.com/paper/1706.08500/) (2017)
- [CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation](https://mlsanity.com/paper/2406.02509/) (2024)
- [A Spatio-temporal Transformer for 3D Human Motion Prediction](https://mlsanity.com/paper/2004.08692/) (2020)
- [CamI2V: Camera-Controlled Image-to-Video Diffusion Model](https://mlsanity.com/paper/2410.15957/) (2024)
- [Vision in Action: Learning Active Perception from Human Demonstrations](https://mlsanity.com/paper/2506.15666/) (2025)
- [HERO: Human Reaction Generation from Videos](https://mlsanity.com/paper/2503.08270/) (2025)
- [ShotVerse: Advancing Cinematic Camera Control for Text-Driven Multi-Shot Video Creation](https://mlsanity.com/paper/2603.11421/) (2026)
- [Martian World Model: Controllable Video Synthesis with Physically Accurate 3D Reconstructions](https://mlsanity.com/paper/2507.07978/) (2025)

## Abstract (arXiv)

Generating high-quality and controllable camera motion is essential for AI-assisted cinematography, video synthesis, and 3D scene understanding. We introduce TKCAM, a Text- and Keyframe-conditioned CAMera-motion synthesis framework based on generative masked modeling. We represent camera dynamics using a 12-dimensional kinematic feature comprising position, velocity, and a continuous rotation representation and discretize them into hierarchical motion tokens via a Residual Vector Quantizer (RVQ). A two-stage masked transformer architecture then learns to reconstruct and refine these tokens, utilizing explicit self- and cross-attention modules for multimodal conditioning. A central feature of our framework is sparse visual keyframe conditioning: users can provide free-form text prompts together with RGB observations at selected timestamps, which provide temporally localized visual guidance for generating coherent in-between trajectories. Furthermore, to advance evaluation standards, we curate RealEstate10K-Cap, a large-scale text-camera dataset, and establish a cross-domain benchmark with a Universal CLaTr Evaluator. Extensive experiments demonstrate that TKCAM surpasses recent state-of-the-art baselines on Fr'echet distance (FID), text-motion matching scores, and retrieval metrics (R@K), while additional analyses evaluate temporal smoothness and cross-domain generalization. Code is available at https://github.com/linearalgebrayhz/TKCAM.