mlsanityData to 14 Oct 2026

RESQUE: Quantifying Estimator to Task and Distribution Shift for Sustainable Model Reusability

arXiv 2412.15511 ↗First posted 20 Dec 2024cs.LG cs.AI cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

As a strategy for sustainability of deep learning, reusing an existing model by retraining it rather than training a new model from scratch is critical. In this paper, we propose REpresentation Shift QUantifying Estimator (RESQUE), a predictive quantifier to estimate the retraining cost of a model to distributional shifts or change of tasks. It provides a single concise index for an estimate of resources required for retraining the model. Through extensive experiments, we show that RESQUE has a strong correlation with various retraining measures. Our results validate that RESQUE is an effective indicator in terms of epochs, gradient norms, changes of parameter magnitude, energy, and carbon emissions. These measures align well with RESQUE for new tasks, multiple noise types, and varying noise intensities. As a result, RESQUE enables users to make informed decisions for retraining to different tasks/distribution shifts and determine the most cost-effective and sustainable option, allowing for the reuse of a model with a much smaller footprint in the environment. The code for this work is available here: https://github.com/JEKimLab/AAAI2025RESQUE

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 27 of its 58 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

27 of 58 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  6. 2019Invariant Risk Minimizationcited by 1,854
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  8. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  9. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  10. 2019Green AIcited by 627
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  12. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
15 more
  1. 2022Efficient Test-Time Model Adaptation without Forgettingcited by 469
  2. 2018TADAM: Task dependent adaptive metric for improved few-shot learningcited by 408
  3. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 339
  4. 2019A Fourier Perspective on Model Robustness in Computer Visioncited by 288
  5. 2020Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixupcited by 261
  6. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
  7. 2018Generalisation in humans and deep neural networkscited by 237
  8. 2019Adversarial Examples Are a Natural Consequence of Test Error in Noisecited by 114
  9. 2022Test-Time Adaptation via Conjugate Pseudo-labelscited by 98
  10. 2020Estimating Example Difficulty Using Variance of Gradientscited by 97
  11. 2023A Systematic Review of Green AIcited by 77
  12. 2020Gradients as a Measure of Uncertainty in Neural Networkscited by 55
  13. 2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21
  14. 2024Aggregate Representation Measure for Predictive Model Reusabilitycited by 2
  15. 2024Estimating Environmental Cost Throughout Model's Adaptive Life Cyclecited by 2

Built on by

Not cited yet.