mlsanityData to 14 Oct 2026

Cinematic Behavior Transfer via NeRF-based Differentiable Filming

arXiv 2311.17754 ↗First posted 29 Nov 2023cs.CV cs.GR cs.HCcited by 12

Not on the map. Why?

Abstract (arXiv)

In the evolving landscape of digital media and video production, the precise manipulation and reproduction of visual elements like camera movements and character actions are highly desired. Existing SLAM methods face limitations in dynamic scenes and human pose estimation often focuses on 2D projections, neglecting 3D statuses. To address these issues, we first introduce a reverse filming behavior estimation technique. It optimizes camera trajectories by leveraging NeRF as a differentiable renderer and refining SMPL tracks. We then introduce a cinematic transfer pipeline that is able to transfer various shot types to a new 2D video or a 3D virtual environment. The incorporation of 3D engine workflow enables superior rendering and control abilities, which also achieves a higher rating in the user study.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 24 of its 36 references that are in the map. First posted on arXiv 29 Nov 2023.

Builds on

24 of 36 references.

References · 24

  1. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,723
  2. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  3. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  4. 2021DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Camerascited by 687
  5. 2021BARF: Bundle-Adjusting Neural Radiance Fieldscited by 642
  6. 2022ViTPose: Simple Vision Transformer Baselines for Human Pose Estimationcited by 456
  7. 2020INeRF: Inverting Neural Radiance Fields for Pose Estimationcited by 382
  8. 2018Learning 3D Human Dynamics from Videocited by 316
  9. 2023ProPainter: Improving Propagation and Transformer for Video Inpaintingcited by 197
  10. 2020Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Videocited by 163
  11. 2021GNeRF: GAN-based Neural Radiance Field without Posed Cameracited by 153
  12. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 153
12 more
  1. 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 142
  2. 2022Parallel Inversion of Neural Radiance Fields for Robust Pose Estimationcited by 55
  3. 2020A Unified Framework for Shot Type Classification Based on Subject Centric Lenscited by 44
  4. 20204D Human Body Capture from Egocentric Video via 3D Scene Groundingcited by 43
  5. 2022D&D: Learning Human Dynamics from Dynamic Cameracited by 41
  6. 2023FlowCam: Training Generalizable 3D Radiance Fields without Camera Poses via Pixel-Aligned Scene Flowcited by 39
  7. 2023LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFscited by 33
  8. 2022The One Where They Reconstructed 3D Humans and Environments in TV Showscited by 30
  9. 2022BodySLAM: Joint Camera Localisation, Mapping, and Human Motion Trackingcited by 24
  10. 2023Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Productioncited by 21
  11. 2021Tracking People by Predicting 3D Appearance, Location & Posecited by 15
  12. 2023JAWS: Just A Wild Shot for Cinematic Transfer in Neural Radiance Fieldscited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 4; 2026: 2.

12 papers.

  1. 2024HumanVid: Demystifying Training Data for Camera-controllable Human Image Animationcited by 64
  2. 2024Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable Single-stage Image-to-3D Generation and Reconstructioncited by 32
  3. 2024E.T. the Exceptional Trajectories: Text-to-camera-trajectory generation with character awarenesscited by 25
  4. 2024AKiRa: Augmentation Kit on Rays for optical video generationcited by 18
  5. 2025Generative AI for Film Creation: A Survey of Recent Advancescited by 17
  6. 2024DreamCinema: Cinematic Transfer with Free Camera and 3D Charactercited by 9
  7. 2025Pulp Motion: Framing-aware multimodal camera and human motion generationcited by 8
  8. 2026MTAVG-Bench 2.0: Diagnosing Failure Modes of Cinematic Expressiveness in Multi-Talker Audio-Video Generationcited by 4
  9. 2024DanceCamAnimator: Keyframe-Based Controllable 3D Dance Camera Synthesiscited by 3
  10. 2026VERTIGO: Visual Preference Optimization for Cinematic Camera Trajectory Generationcited by 3
  11. 2025LensCraft: Your Professional Virtual Cinematographercited by 2
  12. 2025Can Pose Transfer Models Generate Realistic Human Motion?cited by 1