# MMDT: Decoding the Trustworthiness and Safety of Multimodal Foundation Models

arXiv: https://arxiv.org/abs/2503.14827 · HTML: https://mlsanity.com/paper/2503.14827/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Mar 2025.

## Built on by (most cited first, 14 of 14)

- [Agentic Web: Weaving the Next Web with AI Agents](https://mlsanity.com/paper/2507.21206/) (2025)
- [Cer-Eval: Certifiable and Cost-Efficient Evaluation Framework for LLMs](https://mlsanity.com/paper/2505.03814/) (2025)
- [Synthesize Privacy-Preserving High-Resolution Images via Private Textual Intermediaries](https://mlsanity.com/paper/2506.07555/) (2025)
- [Evaluating Bias in Spoken Dialogue LLMs for Real-World Decisions and Recommendations](https://mlsanity.com/paper/2510.02352/) (2025)
- [HEAL: An Empirical Study on Hallucinations in Embodied Agents Driven by Large Language Models](https://mlsanity.com/paper/2506.15065/) (2025)
- [Can VLMs Recall Factual Associations From Visual References?](https://mlsanity.com/paper/2508.18297/) (2025)
- [PII-VisBench: Evaluating Personally Identifiable Information Safety in Vision Language Models Along a Continuum of Visibility](https://mlsanity.com/paper/2601.05739/) (2026)
- [CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs](https://mlsanity.com/paper/2602.03263/) (2026)
- [SafeRoPE: Risk-specific Head-wise Embedding Rotation for Safe Generation in Rectified Flow Transformers](https://mlsanity.com/paper/2604.01826/) (2026)
- [Automated Safety Benchmarking: A Multi-agent Pipeline for LVLMs](https://mlsanity.com/paper/2601.19507/) (2026)
- [ObCLIP: Oblivious CLoud-Device Hybrid Image Generation with Privacy Preservation](https://mlsanity.com/paper/2510.04153/) (2025)
- [AviationLMM: A Large Multimodal Foundation Model for Civil Aviation](https://mlsanity.com/paper/2601.09105/) (2026)
- [Which private attributes do VLMs agree on and predict well?](https://mlsanity.com/paper/2602.07931/) (2026)
- [Trustworthy AI in Digital Health: A Comprehensive Review of Robustness and Explainability](https://mlsanity.com/paper/2608.02238/) (2026)

## Abstract (arXiv)

Multimodal foundation models (MMFMs) play a crucial role in various applications, including autonomous driving, healthcare, and virtual assistants. However, several studies have revealed vulnerabilities in these models, such as generating unsafe content by text-to-image models. Existing benchmarks on multimodal models either predominantly assess the helpfulness of these models, or only focus on limited perspectives such as fairness and privacy. In this paper, we present the first unified platform, MMDT (Multimodal DecodingTrust), designed to provide a comprehensive safety and trustworthiness evaluation for MMFMs. Our platform assesses models from multiple perspectives, including safety, hallucination, fairness/bias, privacy, adversarial robustness, and out-of-distribution (OOD) generalization. We have designed various evaluation scenarios and red teaming algorithms under different tasks for each perspective to generate challenging data, forming a high-quality benchmark. We evaluate a range of multimodal models using MMDT, and our findings reveal a series of vulnerabilities and areas for improvement across these perspectives. This work introduces the first comprehensive and unique safety and trustworthiness evaluation platform for MMFMs, paving the way for developing safer and more reliable MMFMs and systems. Our platform and benchmark are available at https://mmdecodingtrust.github.io/.