# Attention Beats Concatenation for Conditioning Neural Fields

arXiv: https://arxiv.org/abs/2209.10684 · HTML: https://mlsanity.com/paper/2209.10684/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Learning Implicit Fields for Generative Shape Modeling](https://mlsanity.com/paper/1812.02822/) (2018)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2012.00926/) (2020)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations](https://mlsanity.com/paper/2111.13152/) (2021)
- [COTR: Correspondence Transformer for Matching Across Images](https://mlsanity.com/paper/2103.14167/) (2021)
- [CodeNeRF: Disentangled Neural Radiance Fields for Object Categories](https://mlsanity.com/paper/2109.01750/) (2021)
- [Optimizing the Latent Space of Generative Networks](https://mlsanity.com/paper/1707.05776/) (2017)
- [Frequency Bias in Neural Networks for Input of Non-Uniform Density](https://mlsanity.com/paper/2003.04560/) (2020)

## Built on by (most cited first, 20 of 20)

- [ATT3D: Amortized Text-to-3D Object Synthesis](https://mlsanity.com/paper/2306.07349/) (2023)
- [CViT: Continuous Vision Transformer for Operator Learning](https://mlsanity.com/paper/2405.13998/) (2024)
- [INCODE: Implicit Neural Conditioning with Prior Knowledge Embeddings](https://mlsanity.com/paper/2310.18846/) (2023)
- [GINA-3D: Learning to Generate Implicit Neural Assets in the Wild](https://mlsanity.com/paper/2304.02163/) (2023)
- [Variational Autoencoding Neural Operators](https://mlsanity.com/paper/2302.10351/) (2023)
- [ProbNeRF: Uncertainty-Aware Inference of 3D Shapes from 2D Images](https://mlsanity.com/paper/2210.17415/) (2022)
- [Dual Conditioned Diffusion Models for Out-Of-Distribution Detection: Application to Fetal Ultrasound Videos](https://mlsanity.com/paper/2311.00469/) (2023)
- [Geometry-Informed Neural Networks](https://mlsanity.com/paper/2402.14009/) (2024)
- [Vectorized Conditional Neural Fields: A Framework for Solving Time-dependent Parametric Partial Differential Equations](https://mlsanity.com/paper/2406.03919/) (2024)
- [Fast Dynamic 1D Simulation of Divertor Plasmas with Neural PDE Surrogates](https://mlsanity.com/paper/2305.18944/) (2023)
- [HyperNeRFGAN: Hypernetwork approach to 3D NeRF GAN](https://mlsanity.com/paper/2301.11631/) (2023)
- [RENI++ A Rotation-Equivariant, Scale-Invariant, Natural Illumination Prior](https://mlsanity.com/paper/2311.09361/) (2023)
- [Attention Beats Linear for Fast Implicit Neural Representation Generation](https://mlsanity.com/paper/2407.15355/) (2024)
- [Physics-informed Temporal Difference Metric Learning for Robot Motion Planning](https://mlsanity.com/paper/2505.05691/) (2025)
- [DeLiRa: Self-Supervised Depth, Light, and Radiance Fields](https://mlsanity.com/paper/2304.02797/) (2023)
- [TEGLO: High Fidelity Canonical Texture Mapping from Single-View Images](https://mlsanity.com/paper/2303.13743/) (2023)
- [Integrated Object Deformation and Contact Patch Estimation from Visuo-Tactile Feedback](https://mlsanity.com/paper/2305.14470/) (2023)
- [Neural Field Conditioning Strategies for 2D Semantic Segmentation](https://mlsanity.com/paper/2304.14371/) (2023)
- [Diverse Topology Optimization using Modulated Neural Fields](https://mlsanity.com/paper/2502.13174/) (2025)
- [Generative 3D Cardiac Shape Modelling for In-Silico Trials](https://mlsanity.com/paper/2409.16058/) (2024)

## Abstract (arXiv)

Neural fields model signals by mapping coordinate inputs to sampled values. They are becoming an increasingly important backbone architecture across many fields from vision and graphics to biology and astronomy. In this paper, we explore the differences between common conditioning mechanisms within these networks, an essential ingredient in shifting neural fields from memorization of signals to generalization, where the set of signals lying on a manifold is modelled jointly. In particular, we are interested in the scaling behaviour of these mechanisms to increasingly high-dimensional conditioning variables. As we show in our experiments, high-dimensional conditioning is key to modelling complex data distributions, thus it is important to determine what architecture choices best enable this when working on such problems. To this end, we run experiments modelling 2D, 3D, and 4D signals with neural fields, employing concatenation, hyper-network, and attention-based conditioning strategies -- a necessary but laborious effort that has not been performed in the literature. We find that attention-based conditioning outperforms other approaches in a variety of settings.