# Seeing the World through Your Eyes

arXiv: https://arxiv.org/abs/2306.09348 · HTML: https://mlsanity.com/paper/2306.09348/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 62 references that are in the map.
- First posted on arXiv 15 Jun 2023.

## Builds on (most cited first, 30 of 30)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes](https://mlsanity.com/paper/2011.13084/) (2020)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [Dynamic View Synthesis from Dynamic Monocular Video](https://mlsanity.com/paper/2105.06468/) (2021)
- [NoPe-NeRF: Optimising Neural Radiance Field with No Pose Prior](https://mlsanity.com/paper/2212.07388/) (2022)
- [HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields](https://mlsanity.com/paper/2106.13228/) (2021)
- [Robust Dynamic Radiance Fields](https://mlsanity.com/paper/2301.02239/) (2023)
- [NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination](https://mlsanity.com/paper/2106.01970/) (2021)
- [NeRFReN: Neural Radiance Fields with Reflections](https://mlsanity.com/paper/2111.15234/) (2021)
- [Progressively Optimized Local Radiance Fields for Robust View Synthesis](https://mlsanity.com/paper/2303.13791/) (2023)
- [Single Image Reflection Separation with Perceptual Losses](https://mlsanity.com/paper/1806.05376/) (2018)
- [PANDORA: Polarization-Aided Neural Decomposition Of Radiance](https://mlsanity.com/paper/2203.13458/) (2022)
- [Learning Neural Light Fields with Ray-Space Embedding Networks](https://mlsanity.com/paper/2112.01523/) (2021)
- [Non-line-of-Sight Imaging via Neural Transient Fields](https://mlsanity.com/paper/2101.00373/) (2021)
- [Learning to See Through Obstructions](https://mlsanity.com/paper/2004.01180/) (2020)
- [EllSeg: An Ellipse Segmentation Framework for Robust Gaze Tracking](https://mlsanity.com/paper/2007.09600/) (2020)
- [Learning to See Through Obstructions with Layered Decomposition](https://mlsanity.com/paper/2008.04902/) (2020)
- [ORCa: Glossy Objects as Radiance Field Cameras](https://mlsanity.com/paper/2212.04531/) (2022)
- [NLOS-NeuS: Non-line-of-sight Neural Implicit Surface](https://mlsanity.com/paper/2303.12280/) (2023)
- [What You Can Learn by Staring at a Blank Wall](https://mlsanity.com/paper/2108.13027/) (2021)
- [Humans as Light Bulbs: 3D Human Reconstruction from Thermal Reflection](https://mlsanity.com/paper/2305.01652/) (2023)

## Built on by (most cited first, 5 of 5)

- [FlatTrack: Eye-tracking with ultra-thin lensless cameras](https://mlsanity.com/paper/2501.15450/) (2025)
- [Digitally Prototype Your Eye Tracker: Simulating Hardware Performance using 3D Synthetic Data](https://mlsanity.com/paper/2503.16742/) (2025)
- [ViewMask-1-to-3: Multi-View Consistent Image Generation via Multimodal Discrete Diffusion Models](https://mlsanity.com/paper/2512.14099/) (2025)
- [HiFiGaze: Improving Eye Tracking Accuracy Using Screen Content Knowledge](https://mlsanity.com/paper/2603.19588/) (2026)
- [Reflection-aware Generative Novel View Synthesis](https://mlsanity.com/paper/2609.05382/) (2026)

## Abstract (arXiv)

The reflective nature of the human eye is an underappreciated source of information about what the world around us looks like. By imaging the eyes of a moving person, we can collect multiple views of a scene outside the camera's direct line of sight through the reflections in the eyes. In this paper, we reconstruct a 3D scene beyond the camera's line of sight using portrait images containing eye reflections. This task is challenging due to 1) the difficulty of accurately estimating eye poses and 2) the entangled appearance of the eye iris and the scene reflections. Our method jointly refines the cornea poses, the radiance field depicting the scene, and the observer's eye iris texture. We further propose a simple regularization prior on the iris texture pattern to improve reconstruction quality. Through various experiments on synthetic and real-world captures featuring people with varied eye colors, we demonstrate the feasibility of our approach to recover 3D scenes using eye reflections.