# Learning an Ensemble Token from Task-driven Priors in Facial Analysis

arXiv: https://arxiv.org/abs/2507.01290 · HTML: https://mlsanity.com/paper/2507.01290/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 53 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [A Comprehensive Survey of Continual Learning: Theory, Method and Application](https://mlsanity.com/paper/2302.00487/) (2023)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [AffectNet: A Database for Facial Expression, Valence, and Arousal Computing in the Wild](https://mlsanity.com/paper/1708.03985/) (2017)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Visual Transformers: Token-based Image Representation and Processing for Computer Vision](https://mlsanity.com/paper/2006.03677/) (2020)
- [Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Models](https://mlsanity.com/paper/2311.12092/) (2023)
- [Look at Boundary: A Boundary-Aware Face Alignment Algorithm](https://mlsanity.com/paper/1805.10483/) (2018)
- [Implicit Style-Content Separation using B-LoRA](https://mlsanity.com/paper/2403.14572/) (2024)
- [On Memorization in Diffusion Models](https://mlsanity.com/paper/2310.02664/) (2023)
- [RepViT: Revisiting Mobile CNN From ViT Perspective](https://mlsanity.com/paper/2307.09283/) (2023)
- [Wing Loss for Robust Facial Landmark Localisation with Convolutional Neural Networks](https://mlsanity.com/paper/1711.06753/) (2017)
- [MobileOne: An Improved One millisecond Mobile Backbone](https://mlsanity.com/paper/2206.04040/) (2022)
- [LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP](https://mlsanity.com/paper/2404.02285/) (2024)
- [An Adaptive Model Ensemble Adversarial Attack for Boosting Adversarial Transferability](https://mlsanity.com/paper/2308.02897/) (2023)
- [Weighted Ensemble Models Are Strong Continual Learners](https://mlsanity.com/paper/2312.08977/) (2023)
- [FaceXFormer: A Unified Transformer for Facial Analysis](https://mlsanity.com/paper/2403.12960/) (2024)
- [LP-OVOD: Open-Vocabulary Object Detection by Linear Probing](https://mlsanity.com/paper/2310.17109/) (2023)
- [Facial Affective Behavior Analysis with Instruction Tuning](https://mlsanity.com/paper/2404.05052/) (2024)

## Abstract (arXiv)

Facial analysis exhibits task-specific feature variations. While Convolutional Neural Networks (CNNs) have enabled the fine-grained representation of spatial information, Vision Transformers (ViTs) have facilitated the representation of semantic information at the patch level. While advances in backbone architectures have improved over the past decade, combining high-fidelity models often incurs computational costs on feature representation perspective. In this work, we introduce KT-Adapter, a novel methodology for learning knowledge token which enables the integration of high-fidelity feature representation in computationally efficient manner. Specifically, we propose a robust prior unification learning method that generates a knowledge token within a self-attention mechanism, sharing the mutual information across the pre-trained encoders. This knowledge token approach offers high efficiency with negligible computational cost. Our results show improved performance across facial analysis, with statistically significant enhancements observed in the feature representations.