# EigeNet: Geometry-Informed Multi-Modal Learning for Few-shot Novel View RIR Prediction

arXiv: https://arxiv.org/abs/2605.28101 · HTML: https://mlsanity.com/paper/2605.28101/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 42 references that are in the map.
- First posted on arXiv 27 May 2026.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Advancing Open-source World Models](https://mlsanity.com/paper/2601.20540/) (2026)
- [FastVGGT: Training-Free Acceleration of Visual Geometry Transformer](https://mlsanity.com/paper/2509.02560/) (2025)
- [Learning Neural Acoustic Fields](https://mlsanity.com/paper/2204.00628/) (2022)
- [AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesis](https://mlsanity.com/paper/2302.02088/) (2023)
- [Few-Shot Audio-Visual Learning of Environment Acoustics](https://mlsanity.com/paper/2206.04006/) (2022)
- [Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Survey](https://mlsanity.com/paper/2507.14501/) (2025)
- [AV-RIR: Audio-Visual Room Impulse Response Estimation](https://mlsanity.com/paper/2312.00834/) (2023)
- [Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark](https://mlsanity.com/paper/2403.18821/) (2024)
- [FlashVGGT: Efficient and Scalable Visual Geometry Transformers with Compressed Descriptor Attention](https://mlsanity.com/paper/2512.01540/) (2025)
- [Acoustic Volume Rendering for Neural Impulse Response Fields](https://mlsanity.com/paper/2411.06307/) (2024)
- [Hearing Anything Anywhere](https://mlsanity.com/paper/2406.07532/) (2024)
- [Hearing Anywhere in Any Environment](https://mlsanity.com/paper/2504.10746/) (2025)
- [Differentiable Room Acoustic Rendering with Multi-View Vision Priors](https://mlsanity.com/paper/2504.21847/) (2025)
- [Few-shot Acoustic Synthesis with Multimodal Flow Matching](https://mlsanity.com/paper/2603.19176/) (2026)
- [Direction-Aware Neural Acoustic Fields for Few-Shot Interpolation of Ambisonic Impulse Responses](https://mlsanity.com/paper/2505.13617/) (2025)
- [Physics-Informed Direction-Aware Neural Acoustic Fields](https://mlsanity.com/paper/2507.06826/) (2025)
- [Resounding Acoustic Fields with Reciprocity](https://mlsanity.com/paper/2510.20602/) (2025)

## Abstract (arXiv)

Predicting spatially varying Room Impulse Response (RIR) from sparse observations is a critical but highly challenging inverse problem for immersive spatial audio rendering. In this work, we present EIGENET, a geometry-informed multi-modal framework for few-shot novel view RIR prediction. At its core is a Cross-view Alternate-attention Transformer that iteratively refines local intra-view acoustic structures and global cross-view spatial relationships. We empirically demonstrate that this architecture is capable of making full use of the multi-view multi-modal context while performing spatial-temporal reasoning for RIR prediction. Inspired by acoustic ray tracing, we design a geometry-informed modulation block to formulate the connection between geometric features and RIR power spectrum. In the mean time, an auxiliary loss is introduced to transform the single-target waveform prediction into a multi-task learning framework. Through ablation studies, we demonstrate that this design yields consistent performance gains regardless of the underlying backbone, thereby confirming its foundational utility and architecture-agnostic generalizability for RIR prediction task. Evaluated on both simulated and real-world benchmarks, EIGENET achieves both state-of-the-art performance in few-shot novel view RIR prediction and sim-to-real generalization. Codes and checkpoints are available on https://github.com/FEAfeatherTHER/EigeNet.