# Learning A Low-Level Vision Generalist via Visual Task Prompt

arXiv: https://arxiv.org/abs/2408.08601 · HTML: https://mlsanity.com/paper/2408.08601/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 54 references that are in the map.
- First posted on arXiv 16 Aug 2024.

## Builds on (most cited first, 30 of 33)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Restormer: Efficient Transformer for High-Resolution Image Restoration](https://mlsanity.com/paper/2111.09881/) (2021)
- [Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising](https://mlsanity.com/paper/1608.03981/) (2016)
- [Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data](https://mlsanity.com/paper/2107.10833/) (2021)
- [Learning to See in the Dark](https://mlsanity.com/paper/1805.01934/) (2018)
- [Benchmarking Single Image Dehazing and Beyond](https://mlsanity.com/paper/1712.04143/) (2017)
- [Activating More Pixels in Image Super-Resolution Transformer](https://mlsanity.com/paper/2205.04437/) (2022)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform](https://mlsanity.com/paper/1804.02815/) (2018)
- [Deep Joint Rain Detection and Removal from a Single Image](https://mlsanity.com/paper/1609.07769/) (2016)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Multi-Scale Progressive Fusion Network for Single Image Deraining](https://mlsanity.com/paper/2003.10985/) (2020)
- [An Underwater Image Enhancement Benchmark Dataset and Beyond](https://mlsanity.com/paper/1901.05495/) (2019)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [Deep Bilateral Learning for Real-Time Image Enhancement](https://mlsanity.com/paper/1707.02880/) (2017)
- [AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer](https://mlsanity.com/paper/2108.03647/) (2021)
- [Unsupervised Degradation Representation Learning for Blind Super-Resolution](https://mlsanity.com/paper/2104.00416/) (2021)
- [Defocus Deblurring Using Dual-Pixel Data](https://mlsanity.com/paper/2005.00305/) (2020)
- [Compression Artifacts Reduction by a Deep Convolutional Network](https://mlsanity.com/paper/1504.06993/) (2015)
- [Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection](https://mlsanity.com/paper/1909.01955/) (2019)
- [ProRes: Exploring Degradation-aware Visual Prompt for Universal Image Restoration](https://mlsanity.com/paper/2306.13653/) (2023)
- [Conditional Sequential Modulation for Efficient Global Image Retouching](https://mlsanity.com/paper/2009.10390/) (2020)
- [HAT: Hybrid Attention Transformer for Image Restoration](https://mlsanity.com/paper/2309.05239/) (2023)
- [HDRUNet: Single Image HDR Reconstruction with Denoising and Dequantization](https://mlsanity.com/paper/2105.13084/) (2021)

## Built on by (most cited first, 18 of 18)

- [An Intelligent Agentic System for Complex Image Restoration Problems](https://mlsanity.com/paper/2410.17809/) (2024)
- [Factuality Matters: When Image Generation and Editing Meet Structured Visuals](https://mlsanity.com/paper/2510.05091/) (2025)
- [Learning Continuous Wasserstein Barycenter Space for Generalized All-in-One Image Restoration](https://mlsanity.com/paper/2602.23169/) (2026)
- [Lumina-OmniLV: A Unified Multimodal Framework for General Low-Level Vision](https://mlsanity.com/paper/2504.04903/) (2025)
- [WeatherGFM: Learning A Weather Generalist Foundation Model via In-context Learning](https://mlsanity.com/paper/2411.05420/) (2024)
- [FoundIR-v2: Optimizing Pre-Training Data Mixtures for Image Restoration Foundation Model](https://mlsanity.com/paper/2512.09282/) (2025)
- [A Preliminary Exploration Towards General Image Restoration](https://mlsanity.com/paper/2408.15143/) (2024)
- [AllRestorer: All-in-One Transformer for Image Restoration under Composite Degradations](https://mlsanity.com/paper/2411.10708/) (2024)
- [DiffSR: Learning Radar Reflectivity Synthesis via Diffusion Model from Satellite Observations](https://mlsanity.com/paper/2411.06714/) (2024)
- [Exploring Scalable Unified Modeling for General Low-Level Vision](https://mlsanity.com/paper/2507.14801/) (2025)
- [FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration](https://mlsanity.com/paper/2511.14099/) (2025)
- [Position: Agentic Systems Constitute a Key Component of Next-Generation Intelligent Image Processing](https://mlsanity.com/paper/2505.16007/) (2025)
- [BaryIR: Learning Multi-Source Unified Representation in Continuous Barycenter Space for Generalizable All-in-One Image Restoration](https://mlsanity.com/paper/2505.21637/) (2025)
- [Omni-Weather: A Unified Multimodal Model for Weather Radar Understanding and Generation](https://mlsanity.com/paper/2512.21643/) (2025)
- [Degradation-Aware Prompt Learning with Cross-Modal Compensation for Adverse Weather Removal](https://mlsanity.com/paper/2608.06939/) (2026)
- [Vision Generalist Model: A Survey](https://mlsanity.com/paper/2506.09954/) (2025)
- [Degradation-Aware Adaptive Context Gating for Unified Image Restoration](https://mlsanity.com/paper/2605.01236/) (2026)
- [Hidden-Shot: Towards One-Shot Task Generalization for Low-Level Vision Generalist Models](https://mlsanity.com/paper/2607.01535/) (2026)

## Abstract (arXiv)

Building a unified model for general low-level vision tasks holds significant research and practical value. Current methods encounter several critical issues. Multi-task restoration approaches can address multiple degradation-to-clean restoration tasks, while their applicability to tasks with different target domains (e.g., image stylization) is limited. Methods like PromptGIP can handle multiple input-target domains but rely on the Masked Autoencoder (MAE) paradigm. Consequently, they are tied to the ViT architecture, resulting in suboptimal image reconstruction quality. In addition, these methods are sensitive to prompt image content and often struggle with low-frequency information processing. In this paper, we propose a Visual task Prompt-based Image Processing (VPIP) framework to overcome these challenges. VPIP employs visual task prompts to manage tasks with different input-target domains and allows flexible selection of backbone network suitable for general tasks. Besides, a new prompt cross-attention is introduced to facilitate interaction between the input and prompt information. Based on the VPIP framework, we train a low-level vision generalist model, namely GenLV, on 30 diverse tasks. Experimental results show that GenLV can successfully address a variety of low-level tasks, significantly outperforming existing methods both quantitatively and qualitatively. Codes are available at https://github.com/chxy95/GenLV.