Generative World Renderer
Not on the map. Why?
Abstract (arXiv)
Scaling generative inverse and forward rendering to real-world scenarios is bottlenecked by the limited realism and temporal coherence of existing synthetic datasets. To bridge this persistent domain gap, we introduce a large-scale, dynamic dataset curated from visually complex AAA games. Using a novel dual-screen stitched capture method, we extracted 4M continuous frames (720p/30 FPS) of synchronized RGB and five G-buffer channels across diverse scenes, visual effects, and environments, including adverse weather and motion-blur variants. This dataset uniquely advances bidirectional rendering: enabling robust in-the-wild geometry and material decomposition, and facilitating high-fidelity G-buffer-guided video generation. Furthermore, to evaluate the real-world performance of inverse rendering without ground truth, we propose a novel VLM-based assessment protocol measuring semantic, spatial, and temporal consistency. Experiments demonstrate that inverse renderers fine-tuned on our data achieve superior cross-dataset generalization and controllable generation, while our VLM evaluation strongly correlates with human judgment. Combined with our toolkit, our forward renderer enables users to edit styles of AAA games from G-buffers using text prompts.
Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 128 of its 163 references that are in the map. First posted on arXiv 2 Apr 2026.
Builds on
128 of 163 references.
References · 128
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2025Qwen3-VL Technical Reportcited by 2,793
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
- 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
- 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
- 2024Depth Anything V2cited by 1,422
38 more
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
- 2022Exploring CLIP for Assessing the Look and Feel of Imagescited by 991
- 2016Playing for Data: Ground Truth from Computer Gamescited by 818
- 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
- 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
- 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
- 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 537
- 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 523
- 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 506
- 2023Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levelscited by 503
- 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
- 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
- 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
- 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 404
- 2024Metric3Dv2: A Versatile Monocular Geometric Foundation Model for Zero-shot Metric Depth and Surface Normal Estimationcited by 395
- 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 372
- 2021Advances in Neural Renderingcited by 365
- 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
- 2021Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scanscited by 336
- 2023Repurposing Diffusion-Based Image Generators for Monocular Depth Estimationcited by 327
- 2017Super SloMo: High Quality Estimation of Multiple Intermediate Frames for Video Interpolationcited by 326
- 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 326
- 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 300
- 2018Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomizationcited by 294
- 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 286
- 2018DeepMVS: Learning Multi-view Stereopsiscited by 277
- 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 276
- 2018Learning Blind Video Temporal Consistencycited by 253
- 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translationcited by 253
- 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
- 2023GS-IR: 3D Gaussian Splatting for Inverse Renderingcited by 224
- 2020Consistent Video Depth Estimationcited by 220
- 2020Real-Time Intermediate Flow Estimation for Video Frame Interpolationcited by 213
- 2025Video Depth Anything: Consistent Depth Estimation for Super-Long Videoscited by 207
- 2024DepthCrafter: Generating Consistent Long Depth Sequences for Open-world Videoscited by 204
- 2019Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF from a Single Imagecited by 203
- 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
- 2023MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyondcited by 185
50 most cited of 128 All 128 (JSON)
Built on by
Citing papers per year: 2026: 10.
10 papers.
- 2026Programmable World Modelcited by 10
- 2026Generative World Renderer at the Speed of Playcited by 4
- 2026AlayaVista: Streaming World Modeling from Panoramic States to Perspective Videocited by 3
- 2026AI for Games in the Foundation Model Eracited by 3
- 2026WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotationscited by 2
- 2026Oneira: From Open-Ended Generation to Open-World Interaction in Video World Modelscited by 1
- 2026ROWBench: Do Video Models Render What the Program Specifies?cited by 0
- 2026One Figure, Every Canvas: Editable Flowchart Relayout via Agentic Pipelinecited by 0
- 2026AgentGarten: Code Worlds for Evolving Agentscited by 0
- 2026Spectral World Renderer: Unified 3D Hyperspectral Unmixingcited by 0
- 2026Spectral World Renderer: Unified 3D Hyperspectral Unmixing14 Oct 2026
- 2026AgentGarten: Code Worlds for Evolving Agents8 Oct 2026
- 2026One Figure, Every Canvas: Editable Flowchart Relayout via Agentic Pipeline5 Oct 2026
- 2026ROWBench: Do Video Models Render What the Program Specifies?1 Oct 2026
- 2026Oneira: From Open-Ended Generation to Open-World Interaction in Video World Models1 Oct 2026
- 2026AI for Games in the Foundation Model Era15 Sep 2026
- 2026AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video13 Sep 2026
- 2026Programmable World Model9 Sep 2026
- 2026WorldRover: A Scalable Synthetic Video Data Engine for World Exploration with Rich Annotations16 Aug 2026
- 2026Generative World Renderer at the Speed of Play21 Jul 2026
Similar, not cited
Close in wording, no citation either way
- 2026RGBX-Next: Towards Realistic Generative Rendering from G-Buffers
- 2026DreamX-World 1.0: A General-Purpose Interactive World Model
- 2026Closing the Loop: Training-Free Revisit Consistency for Autoregressive Generative Rendering
- 2026SpatialCrafter: Single Image World Modeling with Generative 3D Proxies
- 2026A Hybrid Approach for Closing the Sim2real Appearance Gap in Game Engine Synthetic Datasets
- 2026AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
- 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory
- 2026Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation