# Restore What Matters: Lessons from Joint Restoration and Recognition

arXiv: https://arxiv.org/abs/2609.13791 · HTML: https://mlsanity.com/paper/2609.13791/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 39 of its 86 references that are in the map.
- First posted on arXiv 12 Sep 2026.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)
- [Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring](https://mlsanity.com/paper/1612.02177/) (2016)
- [MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition](https://mlsanity.com/paper/1607.08221/) (2016)
- [The Perception-Distortion Tradeoff](https://mlsanity.com/paper/1711.06077/) (2017)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [Towards Real-World Blind Face Restoration with Generative Facial Prior](https://mlsanity.com/paper/2101.04061/) (2021)
- [CurricularFace: Adaptive Curriculum Learning Loss for Deep Face Recognition](https://mlsanity.com/paper/2004.00288/) (2020)
- [Implicit Diffusion Models for Continuous Super-Resolution](https://mlsanity.com/paper/2303.16491/) (2023)
- [Person Re-identification by Contour Sketch under Moderate Clothing Change](https://mlsanity.com/paper/2002.02295/) (2020)
- [End-to-End Multimodal Emotion Recognition using Deep Neural Networks](https://mlsanity.com/paper/1704.08619/) (2017)
- [Transferable Joint Attribute-Identity Deep Learning for Unsupervised Person Re-Identification](https://mlsanity.com/paper/1803.09786/) (2018)
- [Expanding Accurate Person Recognition to New Altitudes and Ranges: The BRIAR Dataset](https://mlsanity.com/paper/2211.01917/) (2022)
- [Accelerating Atmospheric Turbulence Simulation via Learned Phase-to-Space Transform](https://mlsanity.com/paper/2107.11627/) (2021)
- [WebFace260M: A Benchmark for Million-Scale Deep Face Recognition](https://mlsanity.com/paper/2204.10149/) (2022)
- [KeyPoint Relative Position Encoding for Face Recognition](https://mlsanity.com/paper/2403.14852/) (2024)
- [Single Frame Atmospheric Turbulence Mitigation: A Benchmark Study and A New Physics-Inspired Transformer Model](https://mlsanity.com/paper/2207.10040/) (2022)
- [Imaging through the Atmosphere using Turbulence Mitigation Transformer](https://mlsanity.com/paper/2207.06465/) (2022)
- [Simulating Anisoplanatic Turbulence by Sampling Inter-modal and Spatially Correlated Zernike Coefficients](https://mlsanity.com/paper/2004.11210/) (2020)
- [FarSight: A Physics-Driven Whole-Body Biometric System at Large Distance and Altitude](https://mlsanity.com/paper/2306.17206/) (2023)
- [Physics-Driven Turbulence Image Restoration with Stochastic Refinement](https://mlsanity.com/paper/2307.10603/) (2023)
- [Teaching Where to Look: Attention Similarity Knowledge Distillation for Low Resolution Face Recognition](https://mlsanity.com/paper/2209.14498/) (2022)
- [Spatio-Temporal Turbulence Mitigation: A Translational Perspective](https://mlsanity.com/paper/2401.04244/) (2024)
- [LTT-GAN: Looking Through Turbulence by Inverting GANs](https://mlsanity.com/paper/2112.02379/) (2021)
- [Joint Face Image Restoration and Frontalization for Recognition](https://mlsanity.com/paper/2105.09907/) (2021)

## Abstract (arXiv)

Recognition pipelines typically adopt a restore-then-recognize workflow, yet decades of experience show that generating visually pleasing images seldom translates to improved recognition. We propose a Joint Restoration-for-Recognition (JR2) paradigm: restore only what downstream tasks truly require, with task signals dictating where, how much, and whether restoration is necessary. JR2 rests on three pillars: (i) Physics, employing optics-accurate turbulence simulation, extensible to blur and noise, to ground restoration in real image formation; (ii) Neuroscience, drawing on selective attention and neuroplasticity to direct model capacity toward identity-critical regions and frames while bypassing already-clean inputs; and (iii) Vision & Learning, coupling recognition loss end-to-end through restoration and alignment so that low-level edits maximize high-level identity stability. Evaluations on IARPA-BRIAR show consistent improvements (e.g., TAR@0.01% FAR +0.6; FNIR@1% FPIR -2.5), while a quality gate skips ~70% of clean frames, reducing cost. Ablations confirm physics priors enhance realism, joint training prevents catastrophic forgetting, and selective restoration suffices in many cases. We conclude that better-looking images are neither necessary nor sufficient; restoration modules must be task-driven, selective, and physically aware. Code, pretrained models, and recipes are provided for integration.