# ZipMap: Linear-Time Stateful 3D Reconstruction via Test-Time Training

arXiv: https://arxiv.org/abs/2603.04385 · HTML: https://mlsanity.com/paper/2603.04385/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 86 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Builds on (most cited first, 30 of 65)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data](https://mlsanity.com/paper/2111.08897/) (2021)
- [MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion](https://mlsanity.com/paper/2410.03825/) (2024)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Kubric: A scalable dataset generator](https://mlsanity.com/paper/2203.03570/) (2022)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [Learning to (Learn at Test Time): RNNs with Expressive Hidden States](https://mlsanity.com/paper/2407.04620/) (2024)

## Built on by (most cited first, 26 of 26)

- [VGGT-Omega](https://mlsanity.com/paper/2605.15195/) (2026)
- [Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigation](https://mlsanity.com/paper/2604.10098/) (2026)
- [Mem3R: Streaming 3D Reconstruction with Hybrid Memory via Test-Time Training](https://mlsanity.com/paper/2604.07279/) (2026)
- [Feed-Forward 3D Scene Modeling: A Problem-Driven Perspective](https://mlsanity.com/paper/2604.14025/) (2026)
- [Long-tail Internet photo reconstruction](https://mlsanity.com/paper/2604.22714/) (2026)
- [Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory](https://mlsanity.com/paper/2605.17478/) (2026)
- [X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras](https://mlsanity.com/paper/2607.12993/) (2026)
- [R3: 3D Reconstruction via Relative Regression](https://mlsanity.com/paper/2605.26519/) (2026)
- [G3T Up! Gravity Aligned Coordinate Frames Simplify Pointmap Processing](https://mlsanity.com/paper/2605.27372/) (2026)
- [Anchor3R: Streaming 3D Reconstruction with Transient Anchors for Long-Horizon Visual Mapping](https://mlsanity.com/paper/2606.05035/) (2026)
- [Spark3R: Asymmetric Token Reduction Makes Fast Feed-Forward 3D Reconstruction](https://mlsanity.com/paper/2605.06270/) (2026)
- [TurboVGGT: Fast Visual Geometry Reconstruction with Adaptive Alternating Attention](https://mlsanity.com/paper/2605.14315/) (2026)
- [Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction](https://mlsanity.com/paper/2605.16981/) (2026)
- [Compressing History into Memory: Distilling Transformers into Recurrent Transformers](https://mlsanity.com/paper/2606.21562/) (2026)
- [SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers](https://mlsanity.com/paper/2607.03612/) (2026)
- [GeoFF3D: Coordinate-Anchored Feed-Forward Reconstruction for Large-Scale UAV Mapping](https://mlsanity.com/paper/2608.28288/) (2026)
- [Point4D: Long-range 4D Motion Reconstruction](https://mlsanity.com/paper/2609.09145/) (2026)
- [Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers](https://mlsanity.com/paper/2605.23892/) (2026)
- [Graph Sparse Sampling: Breaking the Curse of the Horizon in Continuous MDP Planning](https://mlsanity.com/paper/2607.05359/) (2026)
- [Online Neural Space Time Memory for Dynamic Novel View Synthesis](https://mlsanity.com/paper/2607.15271/) (2026)
- [GeoWeaver: Accurate Long-Sequence 3D Reconstruction via Hierarchical Geometric Assembly](https://mlsanity.com/paper/2608.17389/) (2026)
- [VI3: Grounding Pretrained 3D Foundation Models with Inertial Cues](https://mlsanity.com/paper/2609.03824/) (2026)
- [Adaptive World Memory 3D Foundation Model for Scalable 3D Mapping, Localization, and Rendering](https://mlsanity.com/paper/2609.21502/) (2026)
- [TTTIR: Unlocking Instance-Specific State Evolution via Test-Time Training for Image Restoration](https://mlsanity.com/paper/2609.26151/) (2026)
- [CLoSeR: Closing the Loop for Long-Context Streaming Reconstruction](https://mlsanity.com/paper/2610.01927/) (2026)
- [Slot3R: Set-Associative Spatial Memory for Streaming 3D Reconstruction](https://mlsanity.com/paper/2610.12282/) (2026)

## Abstract (arXiv)

Feed-forward transformer models have driven rapid progress in 3D vision, but state-of-the-art methods such as VGGT and π3 have a computational cost that scales quadratically with the number of input images, making them inefficient when applied to large image collections. Sequential-reconstruction approaches reduce this cost but sacrifice reconstruction quality. We introduce ZipMap, a stateful feed-forward model that achieves linear-time, bidirectional 3D reconstruction while matching or surpassing the accuracy of quadratic-time methods. ZipMap employs test-time training layers to zip an entire image collection into a compact hidden scene state in a single forward pass, enabling reconstruction of over 700 frames in under 10 seconds on a single H100 GPU, more than 20× faster than state-of-the-art methods such as VGGT. Moreover, we demonstrate the benefits of having a stateful representation in real-time scene-state querying and its extension to sequential streaming reconstruction.