mlsanityData to 7 Oct 2026

Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models

arXiv 2410.08174 ↗First posted 10 Oct 2024cs.CL cs.AI cs.LGcited by 18

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (MLLMs) exhibit promising advancements across various tasks, yet they still encounter significant trustworthiness issues. Prior studies apply Split Conformal Prediction (SCP) in language modeling to construct prediction sets with statistical guarantees. However, these methods typically rely on internal model logits or are restricted to multiple-choice settings, which hampers their generalizability and adaptability in dynamic, open-ended environments. In this paper, we introduce TRON, a two-step framework for risk control and assessment, applicable to any MLLM that supports sampling in both open-ended and closed-ended scenarios. TRON comprises two main components: (1) a novel conformal score to sample response sets of minimum size, and (2) a nonconformity score to identify high-quality responses based on self-consistency theory, controlling the error rates by two specific risk levels. Furthermore, we investigate semantic redundancy in prediction sets within open-ended contexts for the first time, leading to a promising evaluation metric for MLLMs based on average set size. Our comprehensive experiments across four Video Question-Answering (VideoQA) datasets utilizing eight MLLMs show that TRON achieves desired error rates bounded by two user-specified risk levels. Additionally, deduplicated prediction sets maintain adaptiveness while being more efficient and stable for risk assessment under different risk levels.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 46 references that are in the map. First posted on arXiv 10 Oct 2024.

Builds on

39 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective26 May
  2. 2023Multimodal Large Language Models: A Survey22 Nov
  3. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep

Other references · 36

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  7. 2023A Survey of Large Language Modelscited by 2,537
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  10. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  11. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  12. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
24 more
  1. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  2. 2018CoQA: A Conversational Question Answering Challengecited by 674
  3. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  4. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  5. 2022Conformal Risk Controlcited by 290
  6. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  7. 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 270
  8. 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 258
  9. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  10. 2023Conformal Language Modelingcited by 163
  11. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  12. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  13. 2024Language Models with Conformal Factuality Guaranteescited by 128
  14. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  15. 2024Large language model validity via enhanced conformal prediction methodscited by 91
  16. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  17. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 68
  18. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 62
  19. 2024Conformal Prediction for Natural Language Processing: A Surveycited by 52
  20. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  21. 2023Non-Exchangeable Conformal Risk Controlcited by 29
  22. 2024Uncertainty-Aware Evaluation for Vision-Language Modelscited by 28
  23. 2024Word-Sequence Entropy: Towards Uncertainty Estimation in Free-Form Medical Question Answering Applications and Beyondcited by 18
  24. 2024Answering from Sure to Uncertain: Uncertainty-Aware Curriculum Learning for Video Question Answeringcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025One Image is Worth a Thousand Words: A Usability Preservable Text-Image Collaborative Erasing Frameworkcited by 14
  2. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  3. 2025TECP: Token-Entropy Conformal Prediction for LLMscited by 5
  4. 2026SafeGround: Know When to Trust GUI Grounding Models via Uncertainty Calibrationcited by 5
  5. 2026Geometry-Calibrated Conformal Abstention for Language Modelscited by 5
  6. 2025LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systemscited by 4
  7. 2025Data-Driven Calibration of Prediction Sets in Large Vision-Language Models Based on Inductive Conformal Predictioncited by 3
  8. 2026Set-Valued Prediction for Large Language Models with Feasibility-Aware Coverage Guaranteescited by 3
  9. 2025Statistical Management of the False Discovery Rate in Medical Instance Segmentation Based on Conformal Risk Controlcited by 2
  10. 2025Conformal Segmentation in Industrial Surface Defect Detection with Statistical Guaranteescited by 2
  11. 2025Conformal Prediction for Indoor Positioning with Correctness Coverage Guaranteescited by 1
  12. 2025Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guaranteescited by 1
6 more
  1. 2026Uncertainty-Aware Abstention in Large Language Models with Provable Alignment Guaranteescited by 1
  2. 2025Calibrated Prediction Set in Fault Detection with Risk Guarantees via Significance Testscited by 0
  3. 2025Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Controlcited by 0
  4. 2025Hierarchical Spatio-Temporal Attention Network with Adaptive Risk-Aware Decision for Forward Collision Warning in Complex Scenarioscited by 0
  5. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0
  6. 2026Target-Checked Reliability Score Refinement for Video Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation
  2. 2026Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Models
  3. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Models
  4. 2023ChEF: A Comprehensive Evaluation Framework for Standardized Assessment of Multimodal Large Language Models
  5. 2026Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models
  6. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios
  7. 2023MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria
  8. 2024LIME: Less Is More for MLLM Evaluation