# IDSelect: A RL-Based Cost-Aware Selection Agent for Video-based Multi-Modal Person Recognition

arXiv: https://arxiv.org/abs/2602.18990 · HTML: https://mlsanity.com/paper/2602.18990/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 36 of its 51 references that are in the map.
- First posted on arXiv 22 Feb 2026.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 30 of 36)

- [BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks](https://mlsanity.com/paper/1709.01686/) (2017)
- [AdaFace: Quality Adaptive Margin for Face Recognition](https://mlsanity.com/paper/2204.00964/) (2022)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [AdaViT: Adaptive Vision Transformers for Efficient Image Recognition](https://mlsanity.com/paper/2111.15668/) (2021)
- [Person Re-identification: Past, Present and Future](https://mlsanity.com/paper/1610.02984/) (2016)
- [Adaptive Neural Networks for Efficient Inference](https://mlsanity.com/paper/1702.07811/) (2017)
- [Resolution Adaptive Networks for Efficient Inference](https://mlsanity.com/paper/2003.07326/) (2020)
- [Clothes-Changing Person Re-identification with RGB Modality Only](https://mlsanity.com/paper/2204.06890/) (2022)
- [OpenGait: Revisiting Gait Recognition Toward Better Practicality](https://mlsanity.com/paper/2211.06597/) (2022)
- [Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning](https://mlsanity.com/paper/2506.09033/) (2025)
- [Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts](https://mlsanity.com/paper/2410.08245/) (2024)
- [Appearance-Preserving 3D Convolution for Video-based Person Re-identification](https://mlsanity.com/paper/2007.08434/) (2020)
- [GaitSet: Cross-view Gait Recognition through Utilizing Gait as a Deep Set](https://mlsanity.com/paper/2102.03247/) (2021)
- [Exploring Deep Models for Practical Gait Recognition](https://mlsanity.com/paper/2303.03301/) (2023)
- [BigGait: Learning Gait Representation You Want by Large Vision Models](https://mlsanity.com/paper/2402.19122/) (2024)
- [On Learning Disentangled Representations for Gait Recognition](https://mlsanity.com/paper/1909.03051/) (2019)
- [TF-CLIP: Learning Text-free CLIP for Video-based Person Re-Identification](https://mlsanity.com/paper/2312.09627/) (2023)
- [KeyPoint Relative Position Encoding for Face Recognition](https://mlsanity.com/paper/2403.14852/) (2024)
- [AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition](https://mlsanity.com/paper/2105.05165/) (2021)
- [Adaptive Graph Representation Learning for Video Person Re-identification](https://mlsanity.com/paper/1909.02240/) (2019)
- [Learning Clothing and Pose Invariant 3D Shape Representation for Long-Term Person Re-Identification](https://mlsanity.com/paper/2308.10658/) (2023)
- [EdgeFace: Efficient Face Recognition Model for Edge Devices](https://mlsanity.com/paper/2307.01838/) (2023)
- [VarGFaceNet: An Efficient Variable Group Convolutional Neural Network for Lightweight Face Recognition](https://mlsanity.com/paper/1910.04985/) (2019)
- [FarSight: A Physics-Driven Whole-Body Biometric System at Large Distance and Altitude](https://mlsanity.com/paper/2306.17206/) (2023)
- [Controllable and Guided Face Synthesis for Unconstrained Face Recognition](https://mlsanity.com/paper/2207.10180/) (2022)
- [Cluster and Aggregate: Face Recognition with Large Probe Set](https://mlsanity.com/paper/2210.10864/) (2022)
- [Multi-direction and Multi-scale Pyramid in Transformer for Video-based Pedestrian Retrieval](https://mlsanity.com/paper/2202.06014/) (2022)
- [SapiensID: Foundation for Human Recognition](https://mlsanity.com/paper/2504.04708/) (2025)
- [Person Recognition at Altitude and Range: Fusion of Face, Body Shape and Gait](https://mlsanity.com/paper/2505.04616/) (2025)
- [Video Person Re-ID: Fantastic Techniques and Where to Find Them](https://mlsanity.com/paper/1912.05295/) (2019)

## Built on by (most cited first, 1 of 1)

- [FusionAgent: A Multimodal Agent with Dynamic Model Selection for Human Recognition](https://mlsanity.com/paper/2603.26908/) (2026)

## Abstract (arXiv)

Video-based person recognition achieves robust identification by integrating face, body, and gait. However, current systems waste computational resources by processing all modalities with fixed heavyweight ensembles regardless of input complexity. To address these limitations, we propose IDSelect, a reinforcement learning-based cost-aware selector that chooses one pre-trained model per modality per-sequence to optimize the accuracy-efficiency trade-off. Our key insight is that an input-conditioned selector can discover complementary model choices that surpass fixed ensembles while using substantially fewer resources. IDSelect trains a lightweight agent end-to-end using actor-critic reinforcement learning with budget-aware optimization. The reward balances recognition accuracy with computational cost, while entropy regularization prevents premature convergence. At inference, the policy selects the most probable model per modality and fuses modality-specific similarities for the final score. Extensive experiments on challenging video-based datasets demonstrate IDSelect's superior efficiency: on CCVID, it achieves 95.9% Rank-1 accuracy with 92.4% less computation than strong baselines while improving accuracy by 1.8%; on MEVID, it reduces computation by 41.3% while maintaining competitive performance.