# When a Zero-Shooter Cheats: Improving Age Estimation via Activation Steering

arXiv: https://arxiv.org/abs/2605.17658 · HTML: https://mlsanity.com/paper/2605.17658/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Safety alignment, refusal and guardrails, one of 12 directions in LLM safety and security.
- Builds on 22 of its 35 references that are in the map.
- First posted on arXiv 17 May 2026.

## Placed in

- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 22 of 22)

- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark](https://mlsanity.com/paper/2306.06687/) (2023)
- [On the Robustness of Large Multimodal Models Against Image Adversarial Attacks](https://mlsanity.com/paper/2312.03777/) (2023)
- [RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning](https://mlsanity.com/paper/2401.08326/) (2024)
- [MiVOLO: Multi-input Transformer for Age and Gender Estimation](https://mlsanity.com/paper/2307.04616/) (2023)
- [Analysing the Robustness of Vision-Language-Models to Common Corruptions](https://mlsanity.com/paper/2504.13690/) (2025)
- [Zero-Shot Visual Reasoning by Vision-Language Models: Benchmarking and Analysis](https://mlsanity.com/paper/2409.00106/) (2024)
- [Zero-shot Benchmarking: A Framework for Flexible and Scalable Automatic Evaluation of Language Models](https://mlsanity.com/paper/2504.01001/) (2025)
- [Exploring the Limits of Zero Shot Vision Language Models for Hate Meme Detection: The Vulnerabilities and their Interpretations](https://mlsanity.com/paper/2402.12198/) (2024)
- [Double Visual Defense: Adversarial Pre-training and Instruction Tuning for Improving Vision-Language Model Robustness](https://mlsanity.com/paper/2501.09446/) (2025)
- [Beyond Specialization: Assessing the Capabilities of MLLMs in Age and Gender Estimation](https://mlsanity.com/paper/2403.02302/) (2024)
- [Out of the box age estimation through facial imagery: A Comprehensive Benchmark of Vision-Language Models vs. out-of-the-box Traditional Architectures](https://mlsanity.com/paper/2602.07815/) (2026)
- [Evaluating Concept Filtering Defenses against Child Sexual Abuse Material Generation by Text-to-Image Models](https://mlsanity.com/paper/2512.05707/) (2025)
- [Underage Detection through a Multi-Task and MultiAge Approach for Screening Minors in Unconstrained Imagery](https://mlsanity.com/paper/2506.10689/) (2025)

## Abstract (arXiv)

Different age-related regulations have been proposed to protect minors from harmful content and interactions online. Automated age estimation is central to enforcing such regulations, and vision-language models (VLMs) achieve state-of-the-art performance on this task. However, we find that the zero-shot nature of VLM-based age estimation produces an unexpected side effect we call the identity shortcut: Instead of estimating age from visual features, VLMs tend to identify the depicted person and infer their age from memorized knowledge. This phenomenon leads to substantially incorrect predictions when non-celebrities are misidentified as celebrities. It also produces deceptively high robustness to noise and adversarial perturbations on celebrity images, which dominate popular benchmarks. To mitigate this, we propose an activation steering method that suppresses the shortcut by intervening on the hidden states of the VLM. This method improves age estimation accuracy for both memorized and unseen identities, reducing mean absolute error by up to 25% across popular benchmarks.