# T4DT: Tensorizing Time for Learning Temporal 3D Visual Data

arXiv: https://arxiv.org/abs/2208.01421 · HTML: https://mlsanity.com/paper/2208.01421/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 32 references that are in the map.
- First posted on arXiv 2 Aug 2022.

## Builds on (most cited first, 6 of 6)

- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Kaolin: A PyTorch Library for Accelerating 3D Deep Learning Research](https://mlsanity.com/paper/1911.05063/) (2019)
- [TTHRESH: Tensor Compression for Multidimensional Visual Data](https://mlsanity.com/paper/1806.05952/) (2018)
- [Gradient-SDF: A Semi-Implicit Surface Representation for 3D Reconstruction](https://mlsanity.com/paper/2111.13652/) (2021)
- [tntorch: Tensor Network Learning with PyTorch](https://mlsanity.com/paper/2206.11128/) (2022)

## Built on by (most cited first, 2 of 2)

- [One is All: Bridging the Gap Between Neural Radiance Fields Architectures with Progressive Volume Distillation](https://mlsanity.com/paper/2211.15977/) (2022)
- [Progressive Volume Distillation with Active Learning for Efficient NeRF Architecture Conversion](https://mlsanity.com/paper/2304.04012/) (2023)

## Abstract (arXiv)

Unlike 2D raster images, there is no single dominant representation for 3D visual data processing. Different formats like point clouds, meshes, or implicit functions each have their strengths and weaknesses. Still, grid representations such as signed distance functions have attractive properties also in 3D. In particular, they offer constant-time random access and are eminently suitable for modern machine learning. Unfortunately, the storage size of a grid grows exponentially with its dimension. Hence they often exceed memory limits even at moderate resolution. This work proposes using low-rank tensor formats, including the Tucker, tensor train, and quantics tensor train decompositions, to compress time-varying 3D data. Our method iteratively computes, voxelizes, and compresses each frame's truncated signed distance function and applies tensor rank truncation to condense all frames into a single, compressed tensor that represents the entire 4D scene. We show that low-rank tensor compression is extremely compact to store and query time-varying signed distance functions. It significantly reduces the memory footprint of 4D scenes while remarkably preserving their geometric quality. Unlike existing, iterative learning-based approaches like DeepSDF and NeRF, our method uses a closed-form algorithm with theoretical guarantees.