# From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models

arXiv: https://arxiv.org/abs/2607.06553 · HTML: https://mlsanity.com/paper/2607.06553/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 49 references that are in the map.
- First posted on arXiv 7 Jul 2026.

## Builds on (most cited first, 30 of 42)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Deep High-Resolution Representation Learning for Human Pose Estimation](https://mlsanity.com/paper/1902.09212/) (2019)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans](https://mlsanity.com/paper/2110.04994/) (2021)
- [Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation](https://mlsanity.com/paper/2312.02145/) (2023)
- [GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image](https://mlsanity.com/paper/2403.12013/) (2024)
- [Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer](https://mlsanity.com/paper/2511.22699/) (2025)
- [Seedream 4.0: Toward Next-generation Multimodal Image Generation](https://mlsanity.com/paper/2509.20427/) (2025)
- [Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction](https://mlsanity.com/paper/2409.18124/) (2024)
- [PolyFormer: Referring Image Segmentation as Sequential Polygon Generation](https://mlsanity.com/paper/2302.07387/) (2023)
- [Salient Object Detection via Integrity Learning](https://mlsanity.com/paper/2101.07663/) (2021)
- [MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition](https://mlsanity.com/paper/2011.11961/) (2020)
- [OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding](https://mlsanity.com/paper/2406.19389/) (2024)
- [Rethinking Inductive Biases for Surface Normal Estimation](https://mlsanity.com/paper/2403.00712/) (2024)
- [PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model](https://mlsanity.com/paper/2403.14598/) (2024)
- [What Matters When Repurposing Diffusion Models for General Dense Perception Tasks?](https://mlsanity.com/paper/2403.06090/) (2024)
- [Semantic Human Matting](https://mlsanity.com/paper/1809.01354/) (2018)
- [Text4Seg: Reimagining Image Segmentation as Text Generation](https://mlsanity.com/paper/2410.09855/) (2024)
- [Privacy-Preserving Portrait Matting](https://mlsanity.com/paper/2104.14222/) (2021)

## Built on by (most cited first, 1 of 1)

- [Beyond Future Prediction: Denoising as Generative Adaptation for Robot Control](https://mlsanity.com/paper/2609.28339/) (2026)

## Abstract (arXiv)

Large-scale text-to-image models are attractive backbones for dense prediction because RGB generation pretraining learns rich semantic, structural, and geometric priors. Existing generative and editing approaches reuse these priors by casting dense prediction as target generation: annotations such as depth, normals, alpha mattes, masks, and heatmaps are encoded into an RGB-trained VAE latent space and decoded back as image-like targets. We argue this inherits more of the generative output interface than dense prediction requires: unlike RGB synthesis, dense prediction asks for pixel-correct, task-native fields on the same image plane, not new RGB content to be rendered. Our key observation is that a pretrained DiT already organizes RGB inputs through a patch-to-token-to-patch lattice on the image plane, so each token indexes a fixed output patch whose channels can carry task-native quantities instead of RGB appearance. We instantiate this as ReChannel: we keep the VAE encoder for the DiT's input distribution but drop the target-side decoder, adapt the frozen DiT with task LoRA, and map each token to its p x p x K_t pixel-space patch through a shared token-local linear head--about 33K parameters, no spatial mixing. Using FLUX-Klein, we evaluate on six dense prediction tasks and over a dozen benchmarks. This minimal interface sets new state-of-the-art on trimap-free matting, KITTI depth, and referring segmentation, and stays competitive on normals, saliency, and pose. In a matched 4B setting it is more accurate and 2.48x faster than an edit-plus-latent-decode counterpart--dense perception can benefit from generative pretraining without inheriting its output interface.