# Can We Get Rid of Handcrafted Feature Extractors? SparseViT: Nonsemantics-Centered, Parameter-Efficient Image Manipulation Localization through Spare-Coding Transformer

arXiv: https://arxiv.org/abs/2412.14598 · HTML: https://mlsanity.com/paper/2412.14598/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 42 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [IML-ViT: Benchmarking Image Manipulation Localization by Vision Transformer](https://mlsanity.com/paper/2307.14863/) (2023)
- [IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization](https://mlsanity.com/paper/2406.10580/) (2024)

## Built on by (most cited first, 29 of 29)

- [ForensicHub: A Unified Benchmark & Codebase for All-Domain Fake Image Detection and Localization](https://mlsanity.com/paper/2505.11003/) (2025)
- [Venus-DeFakerOne: Unified Fake Image Detection & Localization](https://mlsanity.com/paper/2605.14091/) (2026)
- [AnimeDL-2M: Million-Scale AI-Generated Anime Image Detection and Localization in Diffusion Era](https://mlsanity.com/paper/2504.11015/) (2025)
- [Beyond Fully Supervised Pixel Annotations: Scribble-Driven Weakly-Supervised Framework for Image Manipulation Localization](https://mlsanity.com/paper/2507.13018/) (2025)
- [Webly-Supervised Image Manipulation Localization via Category-Aware Auto-Annotation](https://mlsanity.com/paper/2508.20987/) (2025)
- [Revisiting Image Manipulation Localization under Realistic Manipulation Scenarios](https://mlsanity.com/paper/2509.20006/) (2025)
- [A Lightweight and Effective Image Tampering Localization Network with Vision Mamba](https://mlsanity.com/paper/2502.09941/) (2025)
- [Context-Aware Weakly Supervised Image Manipulation Localization with SAM Refinement](https://mlsanity.com/paper/2503.20294/) (2025)
- [Bridging Semantic Logic Gaps: A Cognition Inspired Multimodal Boundary Preserving Network for Image Manipulation Localization](https://mlsanity.com/paper/2508.07216/) (2025)
- [Multi-modal and Multi-view Fundus Image Fusion for Retinopathy Diagnosis via Multi-scale Cross-attention and Shifted Window Self-attention](https://mlsanity.com/paper/2504.09106/) (2025)
- [Unmasking Synthetic Realities in Generative AI: A Comprehensive Review of Adversarially Robust Deepfake Detection Systems](https://mlsanity.com/paper/2507.21157/) (2025)
- [RelayFormer: A Unified Local-Global Attention Framework for Scalable Image and Video Manipulation Localization](https://mlsanity.com/paper/2508.09459/) (2025)
- [DinoLizer: Separating VAE and Diffusion Artifacts in Generative Inpainting Localization](https://mlsanity.com/paper/2511.20722/) (2025)
- [OmniFD: A Unified Model for Versatile Face Forgery Detection](https://mlsanity.com/paper/2512.01128/) (2025)
- [SAPL: Semantic-Agnostic Prompt Learning in CLIP for Weakly Supervised Image Manipulation Localization](https://mlsanity.com/paper/2601.06222/) (2026)
- [BioTamperNet: Affinity-Guided State-Space Model Detecting Tampered Biomedical Images](https://mlsanity.com/paper/2602.01435/) (2026)
- [Can Image Splicing and Copy-Move Forgery Be Detected by the Same Model? Forensim: An Attention-Based State-Space Approach](https://mlsanity.com/paper/2602.10079/) (2026)
- [PromptForge-350k: A Large-Scale Dataset and Contrastive Framework for Prompt-Based AI Image Forgery Localization](https://mlsanity.com/paper/2603.29386/) (2026)
- [Off-the-shelf Vision Models Benefit Image Manipulation Localization](https://mlsanity.com/paper/2604.09096/) (2026)
- [The Courtroom Trial of Pixels: Robust Image Manipulation Localization via Adversarial Evidence and Reinforcement Learning Judgment](https://mlsanity.com/paper/2604.14703/) (2026)
- [MEDN: Motion-Emotion Feature Decoupling Network for Micro-Expression Recognition](https://mlsanity.com/paper/2604.17899/) (2026)
- [SIGMA: Semantic-Difference Instruction-Grounding Mask Annotator for Text-Driven Image Manipulation Localization](https://mlsanity.com/paper/2605.27924/) (2026)
- [Order within Chaos: Capturing Intrinsic Energy Anomalies for AI-Manipulated Image Forgery Localization](https://mlsanity.com/paper/2606.02178/) (2026)
- [Characterizing Detectability in 3DGS Poisoning: A Stage-wise Benchmark](https://mlsanity.com/paper/2606.03499/) (2026)
- [ForensicsTok: Forensics-Guided Tokenized Modeling for Image Tampering Localization](https://mlsanity.com/paper/2606.24538/) (2026)
- [FakeIDet3-DB: Refining Digital Attacks and Patch Extraction for Secure ID Benchmarking](https://mlsanity.com/paper/2607.26641/) (2026)
- [Evidence-Guided Detection, Localization and Explanation for Text-Centric Image Forensics](https://mlsanity.com/paper/2609.02097/) (2026)
- [From Sharp Eyes to Expert Mind: Internalizing Expert Knowledge in MLLMs for Tampered Text Detection](https://mlsanity.com/paper/2609.36145/) (2026)
- [GIFTBench: Diagnosing Generalization in Image Forgery Localization and Informing Model Design](https://mlsanity.com/paper/2610.01778/) (2026)

## Abstract (arXiv)

Non-semantic features or semantic-agnostic features, which are irrelevant to image context but sensitive to image manipulations, are recognized as evidential to Image Manipulation Localization (IML). Since manual labels are impossible, existing works rely on handcrafted methods to extract non-semantic features. Handcrafted non-semantic features jeopardize IML model's generalization ability in unseen or complex scenarios. Therefore, for IML, the elephant in the room is: How to adaptively extract non-semantic features? Non-semantic features are context-irrelevant and manipulation-sensitive. That is, within an image, they are consistent across patches unless manipulation occurs. Then, spare and discrete interactions among image patches are sufficient for extracting non-semantic features. However, image semantics vary drastically on different patches, requiring dense and continuous interactions among image patches for learning semantic representations. Hence, in this paper, we propose a Sparse Vision Transformer (SparseViT), which reformulates the dense, global self-attention in ViT into a sparse, discrete manner. Such sparse self-attention breaks image semantics and forces SparseViT to adaptively extract non-semantic features for images. Besides, compared with existing IML models, the sparse self-attention mechanism largely reduced the model size (max 80% in FLOPs), achieving stunning parameter efficiency and computation reduction. Extensive experiments demonstrate that, without any handcrafted feature extractors, SparseViT is superior in both generalization and efficiency across benchmark datasets.