# RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination

arXiv: https://arxiv.org/abs/2505.21925 · HTML: https://mlsanity.com/paper/2505.21925/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 54 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions](https://mlsanity.com/paper/2303.12789/) (2023)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Query-Key Normalization for Transformers](https://mlsanity.com/paper/2010.04245/) (2020)
- [Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations](https://mlsanity.com/paper/2111.13152/) (2021)
- [LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias](https://mlsanity.com/paper/2410.17242/) (2024)
- [Liger Kernel: Efficient Triton Kernels for LLM Training](https://mlsanity.com/paper/2410.10989/) (2024)
- [Is Attention All That NeRF Needs?](https://mlsanity.com/paper/2207.13298/) (2022)
- [Real-time Neural Radiance Caching for Path Tracing](https://mlsanity.com/paper/2106.12372/) (2021)
- [Deep Regression on Manifolds: A 3D Rotation Case Study](https://mlsanity.com/paper/2103.16317/) (2021)
- [A Survey on Long Text Modeling with Transformers](https://mlsanity.com/paper/2302.14502/) (2023)
- [RenderNet: A deep convolutional network for differentiable rendering from 3D shapes](https://mlsanity.com/paper/1806.06575/) (2018)
- [Neural Radiosity](https://mlsanity.com/paper/2105.12319/) (2021)
- [ReTR: Modeling Rendering Via Transformer for Generalizable Neural Surface Reconstruction](https://mlsanity.com/paper/2305.18832/) (2023)
- [Neural Voxel Renderer: Learning an Accurate and Controllable Rendering Tool](https://mlsanity.com/paper/1912.04591/) (2019)

## Built on by (most cited first, 23 of 23)

- [Generative World Renderer](https://mlsanity.com/paper/2604.02329/) (2026)
- [Programmable World Model](https://mlsanity.com/paper/2609.10540/) (2026)
- [CUPID: Generative 3D Reconstruction via Joint Object and Pose Modeling](https://mlsanity.com/paper/2510.20776/) (2025)
- [NeAR: Coupled Neural Asset-Renderer Stack](https://mlsanity.com/paper/2511.18600/) (2025)
- [RigidFormer: Learning Rigid Dynamics using Transformers](https://mlsanity.com/paper/2605.09196/) (2026)
- [WorldParticle: Unified World Simulation of Lagrangian Particle Dynamics via Transformer](https://mlsanity.com/paper/2605.15305/) (2026)
- [ClothTransformer: Unified Latent-Space Transformers for Scalable Cloth Simulation](https://mlsanity.com/paper/2605.27852/) (2026)
- [A Generalizable Light Transport 3D Embedding for Global Illumination](https://mlsanity.com/paper/2510.18189/) (2025)
- [RenderFlow: Single-Step Neural Rendering via Flow Matching](https://mlsanity.com/paper/2601.06928/) (2026)
- [RnG: A Unified Transformer for Complete 3D Modeling from Partial Observations](https://mlsanity.com/paper/2603.01194/) (2026)
- [PhysiFormer: Learning to Simulate Mechanics in World Space](https://mlsanity.com/paper/2606.27364/) (2026)
- [Neural Cone Radiosity for Interactive Global Illumination with Glossy Materials](https://mlsanity.com/paper/2509.07522/) (2025)
- [SPRig: Self-Supervised Pose-Invariant Rigging from Mesh Sequences](https://mlsanity.com/paper/2602.12740/) (2026)
- [From Particles to Fields: Reframing Photon Mapping with Continuous Gaussian Photon Fields](https://mlsanity.com/paper/2512.12459/) (2025)
- [A Few-Step Generative Model on Cumulative Flow Maps](https://mlsanity.com/paper/2605.03623/) (2026)
- [TOPOS: High-Fidelity and Efficient Industry-Grade 3D Head Generation](https://mlsanity.com/paper/2605.14594/) (2026)
- [Learning Laplacian Eigenspace with Mass-Aware Neural Operators on Point Clouds](https://mlsanity.com/paper/2605.24390/) (2026)
- [Learning View-Dependent Splatting Kernels](https://mlsanity.com/paper/2605.25426/) (2026)
- [Physically-guided Image Generation for Multi-Projection Mapping](https://mlsanity.com/paper/2606.22477/) (2026)
- [RenderFormer++: Scalable and Physics-Informed Feed-Forward Neural Rendering](https://mlsanity.com/paper/2606.30380/) (2026)
- [ViDS: Video Diffusion Shader using 3D Face Tracking](https://mlsanity.com/paper/2607.24124/) (2026)
- [RenderFormer-V2: Neural Rendering with Heterogeneous Scene Primitives](https://mlsanity.com/paper/2609.05738/) (2026)
- [Physically Based Rendering in the Latent Space](https://mlsanity.com/paper/2609.21054/) (2026)

## Abstract (arXiv)

We present RenderFormer, a neural rendering pipeline that directly renders an image from a triangle-based representation of a scene with full global illumination effects and that does not require per-scene training or fine-tuning. Instead of taking a physics-centric approach to rendering, we formulate rendering as a sequence-to-sequence transformation where a sequence of tokens representing triangles with reflectance properties is converted to a sequence of output tokens representing small patches of pixels. RenderFormer follows a two stage pipeline: a view-independent stage that models triangle-to-triangle light transport, and a view-dependent stage that transforms a token representing a bundle of rays to the corresponding pixel values guided by the triangle-sequence from the view-independent stage. Both stages are based on the transformer architecture and are learned with minimal prior constraints. We demonstrate and evaluate RenderFormer on scenes with varying complexity in shape and light transport.