# Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction

arXiv: https://arxiv.org/abs/2604.08542 · HTML: https://mlsanity.com/paper/2604.08542/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 80 of its 104 references that are in the map.
- First posted on arXiv 9 Apr 2026.

## Builds on (most cited first, 30 of 80)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [Grounding Image Matching in 3D with MASt3R](https://mlsanity.com/paper/2406.09756/) (2024)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes](https://mlsanity.com/paper/2308.11417/) (2023)
- [MegaDepth: Learning Single-View Depth Prediction from Internet Photos](https://mlsanity.com/paper/1804.00607/) (2018)
- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding](https://mlsanity.com/paper/2011.02523/) (2020)
- [LightGlue: Local Feature Matching at Light Speed](https://mlsanity.com/paper/2306.13643/) (2023)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Linear Transformers Are Secretly Fast Weight Programmers](https://mlsanity.com/paper/2102.11174/) (2021)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [VINS-Mono: A Robust and Versatile Monocular Visual-Inertial State Estimator](https://mlsanity.com/paper/1708.03852/) (2017)
- [Parallelizing Linear Transformers with the Delta Rule over Sequence Length](https://mlsanity.com/paper/2406.06484/) (2024)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)

## Built on by (most cited first, 30 of 31)

- [LingBot-Map: Geometric Context Transformer for Streaming 3D Reconstruction](https://mlsanity.com/paper/2604.14141/) (2026)
- [MemoryWAM: Efficient World Action Modeling with Persistent Memory](https://mlsanity.com/paper/2606.20562/) (2026)
- [Long-tail Internet photo reconstruction](https://mlsanity.com/paper/2604.22714/) (2026)
- [Glob3R: Global Structure-from-Motion with 3D Foundation Models](https://mlsanity.com/paper/2607.09225/) (2026)
- [Mamba-VGGT: Persistent Long-Sequence Video Geometry Grounded Transformer via External Sliding Window Mamba Memory](https://mlsanity.com/paper/2605.17478/) (2026)
- [EPS3D: End-to-End Feed-Forward 3D Panoptic Segmentation](https://mlsanity.com/paper/2606.08980/) (2026)
- [X-Lens: Real-Time Metric Depth Estimation with Heterogeneous Cameras](https://mlsanity.com/paper/2607.12993/) (2026)
- [QuerySplat: Decoupling Geometry and Appearance Representations in 3DGS Prediction](https://mlsanity.com/paper/2608.01186/) (2026)
- [R3: 3D Reconstruction via Relative Regression](https://mlsanity.com/paper/2605.26519/) (2026)
- [Anchor3R: Streaming 3D Reconstruction with Transient Anchors for Long-Horizon Visual Mapping](https://mlsanity.com/paper/2606.05035/) (2026)
- [LIST3R: Long-sequence Instance-aware 3D Reconstruction](https://mlsanity.com/paper/2607.00375/) (2026)
- [Rethinking the State Update Gate for Long-Sequence Recurrent 3D Reconstruction](https://mlsanity.com/paper/2605.16981/) (2026)
- [Compressing History into Memory: Distilling Transformers into Recurrent Transformers](https://mlsanity.com/paper/2606.21562/) (2026)
- [RayTun3R: Online Camera Adaptation in 3D Foundation Models](https://mlsanity.com/paper/2607.02711/) (2026)
- [SAF3R: Dynamic Sparse Attention for Feed-Forward 3D Reconstruction Transformers](https://mlsanity.com/paper/2607.03612/) (2026)
- [GeoFF3D: Coordinate-Anchored Feed-Forward Reconstruction for Large-Scale UAV Mapping](https://mlsanity.com/paper/2608.28288/) (2026)
- [LongDPM: Overlap-Aware 4D Reconstruction from Long Monocular Videos](https://mlsanity.com/paper/2605.17303/) (2026)
- [Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers](https://mlsanity.com/paper/2605.23892/) (2026)
- [SpatialBench: Is Your Spatial Foundation Model an All-Round Player?](https://mlsanity.com/paper/2605.27367/) (2026)
- [DrivingDepth: Sparse-Prompted Pixel-wise Scale Correction for Driving Depth Estimation](https://mlsanity.com/paper/2606.31488/) (2026)
- [On-the-Fly3R: Towards Robust Online 3D Reconstruction with Feed-Forward 3R Models for Large-Scale UAV Scenarios](https://mlsanity.com/paper/2609.00923/) (2026)
- [FFVO: A Feedforward Pose Decoder for Long-Horizon Visual Odometry](https://mlsanity.com/paper/2609.13733/) (2026)
- [SkyAnchor: Updating Metric-scale Aerial 3D Gaussian Scenes from Unposed Ground-View Sequences](https://mlsanity.com/paper/2609.13903/) (2026)
- [SURE-Map: Self-Correcting Streaming Geometric Foundation Models](https://mlsanity.com/paper/2609.15795/) (2026)
- [AMB3R-SLAM: Kilometer-scale SLAM with Hierarchical Backend](https://mlsanity.com/paper/2609.19518/) (2026)
- [Adaptive World Memory 3D Foundation Model for Scalable 3D Mapping, Localization, and Rendering](https://mlsanity.com/paper/2609.21502/) (2026)
- [VGGT-Bridge: Beyond Sequential Pose Graphs via Coarse-Stride Skip Edges](https://mlsanity.com/paper/2610.06594/) (2026)
- [Slot3R: Set-Associative Spatial Memory for Streaming 3D Reconstruction](https://mlsanity.com/paper/2610.12282/) (2026)
- [MIRAGE: Metric Inference and Reliability-Aware Gaussian Estimation from Feed-Forward Vision](https://mlsanity.com/paper/2610.12568/) (2026)
- [UniMetric: Universal Metric Scale Recovery for Visual Geometry Foundation Models](https://mlsanity.com/paper/2610.16945/) (2026)

## Abstract (arXiv)

This paper addresses the task of large-scale 3D scene reconstruction from long video sequences. Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints. However, these methods often struggle to maintain reconstruction accuracy and consistency over long sequences due to limited memory capacity and the inability to effectively capture global contextual cues. In contrast, humans can naturally exploit the global understanding of the scene to inform local perception. Motivated by this, we propose a novel neural global context representation that efficiently compresses and retains long-range scene information, enabling the model to leverage extensive contextual cues for enhanced reconstruction accuracy and consistency. The context representation is realized through a set of lightweight neural sub-networks that are rapidly adapted during test time via self-supervised objectives, which substantially increases memory capacity without incurring significant computational overhead. The experiments on multiple large-scale benchmarks, including the KITTI Odometry~citeGeiger2012CVPR and Oxford Spires~citetao2025spires datasets, demonstrate the effectiveness of our approach in handling ultra-large scenes, achieving leading pose accuracy and state-of-the-art 3D reconstruction accuracy while maintaining efficiency. Code is available at https://zju3dv.github.io/scal3r.