# FrameDiffuser: G-Buffer-Conditioned Diffusion for Neural Forward Frame Rendering

arXiv: https://arxiv.org/abs/2512.16670 · HTML: https://mlsanity.com/paper/2512.16670/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 18 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion](https://mlsanity.com/paper/2407.01392/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [RGBleftrightarrowX: Image decomposition and synthesis using material- and lighting-aware diffusion models](https://mlsanity.com/paper/2405.00666/) (2024)
- [StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generation](https://mlsanity.com/paper/2312.12491/) (2023)
- [AR-Diffusion: Asynchronous Video Generation with Auto-Regressive Diffusion](https://mlsanity.com/paper/2503.07418/) (2025)
- [SIGNeRF: Scene Integrated Generation for Neural Radiance Fields](https://mlsanity.com/paper/2401.01647/) (2024)
- [SViM3D: Stable Video Material Diffusion for Single Image 3D Generation](https://mlsanity.com/paper/2510.08271/) (2025)
- [PBR-Inspired Controllable Diffusion for Image Generation](https://mlsanity.com/paper/2503.15147/) (2025)

## Built on by (most cited first, 3 of 3)

- [Generative World Renderer at the Speed of Play](https://mlsanity.com/paper/2607.18703/) (2026)
- [Relit-LiVE: Relight Video by Jointly Learning Environment Video](https://mlsanity.com/paper/2605.06658/) (2026)
- [BRDFusion: Physics Meets Generation for Urban Scene Inverse Rendering](https://mlsanity.com/paper/2606.17049/) (2026)

## Abstract (arXiv)

Neural rendering for interactive applications requires translating geometric and material properties (G-buffer) to photorealistic images with realistic lighting on a frame-by-frame basis. While recent diffusion-based approaches show promise for G-buffer-conditioned image synthesis, they face critical limitations: single-image models like RGBX generate frames independently without temporal consistency, while video models like DiffusionRenderer are too computationally expensive for most consumer gaming sets ups and require complete sequences upfront, making them unsuitable for interactive applications where future frames depend on user input. We introduce FrameDiffuser, an autoregressive neural rendering framework that generates temporally consistent, photorealistic frames by conditioning on G-buffer data and the models own previous output. After an initial frame, FrameDiffuser operates purely on incoming G-buffer data, comprising geometry, materials, and surface properties, while using its previously generated frame for temporal guidance, maintaining stable, temporal consistent generation over hundreds to thousands of frames. Our dual-conditioning architecture combines ControlNet for structural guidance with ControlLoRA for temporal coherence. A three-stage training strategy enables stable autoregressive generation. We specialize our model to individual environments, prioritizing consistency and inference speed over broad generalization, demonstrating that environment-specific training achieves superior photorealistic quality with accurate lighting, shadows, and reflections compared to generalized approaches.