mlsanityData to 14 Oct 2026

Test-Time Training on Video Streams

arXiv 2307.05014 ↗First posted 11 Jul 2023cs.CV cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Prior work has established Test-Time Training (TTT) as a general framework to further improve a trained model at test time. Before making a prediction on each test instance, the model is first trained on the same instance using a self-supervised task such as reconstruction. We extend TTT to the streaming setting, where multiple test instances - video frames in our case - arrive in temporal order. Our extension is online TTT: The current model is initialized from the previous model, then trained on the current frame and a small window of frames immediately before. Online TTT significantly outperforms the fixed-model baseline for four tasks, on three real-world datasets. The improvements are more than 2.2x and 1.5x for instance and panoptic segmentation. Surprisingly, online TTT also outperforms its offline variant that accesses strictly more information, training on all frames from the entire test video regardless of temporal order. This finding challenges those in prior work using synthetic videos. We formalize a notion of locality as the advantage of online over offline TTT, and analyze its role with ablations and a theory based on bias-variance trade-off.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 63 of its 95 references that are in the map. First posted on arXiv 11 Jul 2023.

Builds on

63 of 95 references.

References · 63

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  7. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  8. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  10. 2016Learning without Forgettingcited by 2,496
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
38 more
  1. 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
  2. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  4. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,428
  5. 2017Continual Learning with Deep Generative Replaycited by 1,158
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  7. 2016Colorful Image Colorizationcited by 1,114
  8. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  9. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  10. 2019Three scenarios for continual learningcited by 614
  11. 2014Convex Optimization: Algorithms and Complexitycited by 575
  12. 2019Self-labelling via simultaneous clustering and representation learningcited by 528
  13. 2020Improving robustness against common corruptions by covariate shift adaptationcited by 431
  14. 2018Dynamic Few-Shot Visual Learning without Forgettingcited by 385
  15. 2020Rethinking Pre-training and Self-trainingcited by 352
  16. 2017"Zero-Shot" Super-Resolution using Deep Internal Learningcited by 286
  17. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  18. 2018YouTube-VOS: Sequence-to-Sequence Video Object Segmentationcited by 232
  19. 2018Task-Free Continual Learningcited by 221
  20. 2020Consistent Video Depth Estimationcited by 220
  21. 2022Test-Time Training with Masked Autoencoderscited by 212
  22. 2023Robust Test-Time Adaptation in Dynamic Scenarioscited by 189
  23. 2020Instance Adaptive Self-Training for Unsupervised Domain Adaptationcited by 177
  24. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  25. 2021Self-Supervised Models are Continual Learnerscited by 149
  26. 2020Video Panoptic Segmentationcited by 142
  27. 2017Data Distillation: Towards Omni-Supervised Learningcited by 126
  28. 2017Real-Time User-Guided Image Colorization with Learned Deep Priorscited by 123
  29. 2020Self-Supervised Policy Adaptation during Deploymentcited by 123
  30. 2013Learning by Transductioncited by 121
  31. 2018Don't forget, there is more than forgetting: new metrics for Continual Learningcited by 115
  32. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 100
  33. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  34. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 88
  35. 2022SePiCo: Semantic-Guided Pixel Contrast for Domain Adaptive Semantic Segmentationcited by 87
  36. 2018Real-time self-adaptive deep stereocited by 80
  37. 2019Deep Exemplar-based Video Colorizationcited by 75
  38. 2021STEP: Segmenting and Tracking Every Pixelcited by 73

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 4; 2025: 17; 2026: 12.

37 papers.

Sort:

By area:

  1. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 355
  2. 2025TTRL: Test-Time Reinforcement Learningcited by 275
  3. 2025End-to-End Test-Time Training for Long Contextcited by 62
  4. 2026AdaJEPA: An Adaptive Latent World Modelcited by 20
  5. 2023Learning to (Learn at Test Time)cited by 17
  6. 2026Pathwise Test-Time Correction for Autoregressive Long Video Generationcited by 15
  7. 2025SPINE: Token-Selective Test-Time Reinforcement Learning with Entropy-Band Regularizationcited by 12
  8. 2023Learning from One Continuous Video Streamcited by 11
  9. 2023Appearance-Based Refinement for Object-Centric Motion Segmentationcited by 11
  10. 2024CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Trainingcited by 9
  11. 2025Learning from Streaming Video with Orthogonal Gradientscited by 9
  12. 2025Learning with Calibration: Exploring Test-Time Computing of Spatio-Temporal Forecastingcited by 9
25 more
  1. 2026Tool Verification for Test-Time Reinforcement Learningcited by 9
  2. 2025LoRA-TTT: Low-Rank Test-Time Training for Vision-Language Modelscited by 7
  3. 2026DexWorldModel: Causal Latent World Modeling towards Automated Learning of Embodied Taskscited by 5
  4. 2025TTOM: Test-Time Optimization and Memorization for Compositional Video Generationcited by 4
  5. 2025video-SALMONN S: Memory-Enhanced Streaming Audio-Visual LLMcited by 4
  6. 2023Single Image Test-Time Adaptation for Segmentationcited by 3
  7. 2025GenAnalysis: Joint Shape Analysis by Learning Man-Made Shape Generators with Deformation Regularizationscited by 3
  8. 2025From Invariant Representations to Invariant Data: Provable Robustness to Spurious Correlations via Noisy Counterfactual Matchingcited by 3
  9. 2026Learning Cross-View Object Correspondence via Cycle-Consistent Mask Predictioncited by 3
  10. 2026FOZO: Forward-Only Zeroth-Order Prompt Optimization for Test-Time Adaptationcited by 3
  11. 2024A Modular System for Enhanced Robustness of Multimedia Understanding Networks via Deep Parametric Estimationcited by 2
  12. 2025Event Stream-based Visual Object Tracking: HDETrack V2 and A High-Definition Benchmarkcited by 1
  13. 2025MADCAT: Combating Malware Detection Under Concept Drift with Test-Time Adaptationcited by 1
  14. 2025VITA: Zero-Shot Value Functions via Test-Time Adaptation of Vision-Language Modelscited by 1
  15. 2025An Investigation of Test-time Adaptation for Audio Classification under Background Noisecited by 1
  16. 2024One protein is all you needcited by 0
  17. 2025GISE-TTT:A Framework for Global InformationSegmentation and Enhancementcited by 0
  18. 2025Uncertainty-aware Test-Time Training (UT3) for Efficient On-the-fly Domain Adaptive Dense Regressioncited by 0
  19. 2025Sequence-Adaptive Video Prediction in Continuous Streams using Diffusion Noise Optimizationcited by 0
  20. 2026Precedent-Informed Reasoning: Mitigating Overthinking in Large Reasoning Models via Test-Time Precedent Learningcited by 0
  21. 2026Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streamscited by 0
  22. 2026Forget, Anticipate and Adapt: Test Time Training for Long Videoscited by 0
  23. 2026When Should the Count Change? Learning State Maintenance for Causal Video Countingcited by 0
  24. 2026I Have a Stream: Making Self-Supervised Learning Work on Continuous Videocited by 0
  25. 2026JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shiftscited by 0