mlsanityData to 14 Oct 2026

Conformal Predictions for Human Action Recognition with Vision-Language Models

arXiv 2502.06631 ↗First posted 10 Feb 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Human-in-the-Loop (HITL) systems are essential in high-stakes, real-world applications where AI must collaborate with human decision-makers. This work investigates how Conformal Prediction (CP) techniques, which provide rigorous coverage guarantees, can enhance the reliability of state-of-the-art human action recognition (HAR) systems built upon Vision-Language Models (VLMs). We demonstrate that CP can significantly reduce the average number of candidate classes without modifying the underlying VLM. However, these reductions often result in distributions with long tails which can hinder their practical utility. To mitigate this, we propose tuning the temperature of the softmax prediction, without using additional calibration data. This work contributes to ongoing efforts for multi-modal human-AI interaction in dynamic real-world environments.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 20 of its 32 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

20 of 32 references.

References · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  3. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  4. 2018SlowFast Networks for Video Recognitioncited by 1,867
  5. 2021ViViT: A Video Vision Transformercited by 1,620
  6. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  7. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  8. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  9. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  10. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 272
  11. 2022Fine-tuned CLIP Models are Efficient Video Learnerscited by 179
  12. 2019STM: SpatioTemporal and Motion Encoding for Action Recognitioncited by 142
8 more
  1. 2024EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameterscited by 99
  2. 2013A Conformal Prediction Approach to Explore Functional Datacited by 86
  3. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?cited by 67
  4. 2024A Hard-to-Beat Baseline for Training-free CLIP-based Adaptationcited by 53
  5. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  6. 2019Fast User-Guided Video Object Segmentation by Interaction-and-Propagation Networkscited by 33
  7. 2023Designing Decision Support Systems Using Counterfactual Prediction Setscited by 26
  8. 2024Are foundation models for computer vision good conformal predictors?cited by 13

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025VLA-Adapter: An Effective Paradigm for Tiny-Scale Vision-Language-Action Model
  2. 2026FabriVLA: A Lightweight Vision-Language-Action Model with Conformal Action Chunk Uncertainty
  3. 2026Decision-Aware Uncertainty Evaluation of Vision-Language Model-Based Early Action Anticipation for Human-Robot Interaction
  4. 202510 Open Challenges Steering the Future of Vision-Language-Action Models
  5. 2025RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
  6. 2025A Survey on Efficient Vision-Language-Action Models
  7. 2026Action Draft and Verify: A Self-Verifying Framework for Vision-Language-Action Model
  8. 2026Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization