mlsanityData to 14 Oct 2026

Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback

arXiv 2309.15762 ↗First posted 27 Sep 2023cs.CV cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

We propose a method for adapting neural networks to distribution shifts at test-time. In contrast to training-time robustness mechanisms that attempt to anticipate and counter the shift, we create a closed-loop system and make use of a test-time feedback signal to adapt a network on the fly. We show that this loop can be effectively implemented using a learning-based function, which realizes an amortized optimizer for the network. This leads to an adaptation method, named Rapid Network Adaptation (RNA), that is notably more flexible and orders of magnitude faster than the baselines. Through a broad set of experiments using various adaptation signals and target tasks, we study the efficiency and flexibility of this method. We perform the evaluations using various datasets (Taskonomy, Replica, ScanNet, Hypersim, COCO, ImageNet), tasks (depth, optical flow, semantic segmentation, classification), and distribution shifts (Cross-datasets, 2D and 3D Common Corruptions) with promising results. We end with a discussion on general formulations for handling distribution shifts and our observations from comparing with similar approaches from other domains.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 103 of its 124 references that are in the map. First posted on arXiv 27 Sep 2023.

Builds on

103 of 124 references.

References · 103

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  11. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  12. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
38 more
  1. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2022A ConvNet for the 2020scited by 3,579
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  8. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  9. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  10. 2016Matching Networks for One Shot Learningcited by 2,255
  11. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 2,010
  12. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  13. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  14. 2021Vision Transformers for Dense Predictioncited by 1,803
  15. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  16. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  17. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,495
  18. 2017SuperPoint: Self-Supervised Interest Point Detection and Descriptioncited by 1,320
  19. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  20. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  21. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  22. 2016Group Equivariant Convolutional Networkscited by 1,165
  23. 2016HyperNetworkscited by 1,027
  24. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 904
  25. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  26. 2020Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptationcited by 848
  27. 2019On the Convergence of Adam and Beyondcited by 841
  28. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  29. 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
  30. 2016Learning to learn by gradient descent by gradient descentcited by 620
  31. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 604
  32. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  33. 2017Look, Listen and Learncited by 458
  34. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  35. 2021In-Place Scene Labelling and Understanding with Implicit Scene Representationcited by 415
  36. 2023Towards Stable Test-Time Adaptation in Dynamic Wild Worldcited by 415
  37. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  38. 2021MEMO: Test Time Robustness via Adaptation and Augmentationcited by 377

50 most cited of 103 All 103 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 6; 2026: 2.

10 papers.

  1. 2024Backpropagation-free Network for 3D Test-time Adaptationcited by 14
  2. 2025Free on the Fly: Enhancing Flexibility in Test-Time Adaptation with Online EMcited by 8
  3. 2024Test-Time Adaptation in Point Clouds: Leveraging Sampling Variation with Weight Averagingcited by 7
  4. 2026MMEarth-Bench: Global Model Adaptation via Multimodal Test-Time Trainingcited by 7
  5. 2025Point-Cache: Test-time Dynamic and Hierarchical Cache for Robust and Generalizable Point Cloud Analysiscited by 4
  6. 2025ETA: Energy-based Test-time Adaptation for Depth Completioncited by 4
  7. 2025Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cuescited by 3
  8. 2025Purge-Gate: Backpropagation-Free Test-Time Adaptation for Point Clouds Classification via Token Purgingcited by 3
  9. 2025Low-Cost Test-Time Adaptation for Robust Video Editingcited by 1
  10. 2026SPeaR: Test-Time Adaptation with Steering Primitives for Realigning Representationscited by 0