mlsanityData to 7 Oct 2026

Trustworthy Large Models in Vision: A Survey

arXiv 2311.09680 ↗First posted 16 Nov 2023cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The rapid progress of Large Models (LMs) has recently revolutionized various fields of deep learning with remarkable grades, ranging from Natural Language Processing (NLP) to Computer Vision (CV). However, LMs are increasingly challenged and criticized by academia and industry due to their powerful performance but untrustworthy behavior, which urgently needs to be alleviated by reliable methods. Despite the abundance of literature on trustworthy LMs in NLP, a systematic survey specifically delving into the trustworthiness of LMs in CV remains absent. In order to mitigate this gap, we summarize four relevant concerns that obstruct the trustworthy usage in vision of LMs in this survey, including 1) human misuse, 2) vulnerability, 3) inherent issue and 4) interpretability. By highlighting corresponding challenge, countermeasures, and discussion in each topic, we hope this survey will facilitate readers' understanding of this field, promote alignment of LMs with human expectations and enable trustworthy LMs to serve as welfare rather than disaster for human society.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 188 of its 281 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

188 of 281 references.

References · 188

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  11. 2023Visual Instruction Tuningcited by 7,792
  12. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
38 more
  1. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  13. 2016Deep Learning with Differential Privacycited by 2,682
  14. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  15. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  16. 2020Extracting Training Data from Large Language Modelscited by 2,144
  17. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  18. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  19. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  20. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  21. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  22. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  23. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
  24. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  25. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  26. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  27. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  28. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  29. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  30. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  31. 2018This Looks Like That: Deep Learning for Interpretable Image Recognitioncited by 731
  32. 2023Extracting Training Data from Diffusion Modelscited by 723
  33. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  34. 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 682
  35. 2016Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approachcited by 663
  36. 2018Object Hallucination in Image Captioningcited by 661
  37. 2019Text Summarization with Pretrained Encoderscited by 553
  38. 2016Self-critical Sequence Training for Image Captioningcited by 544

50 most cited of 188 All 188 (JSON)

Built on by

Not cited yet.