mlsanityData to 14 Oct 2026

Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics

arXiv 2309.07120 ↗First posted 13 Sep 2023cs.CL cs.AI cs.CVcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal large language models (MLLMs) are trained based on large language models (LLM), with an enhanced capability to comprehend multi-modal inputs and generate textual responses. While they excel in multi-modal tasks, the pure NLP abilities of MLLMs are often underestimated and left untested. In this study, we get out of the box and unveil an intriguing characteristic of MLLMs -- our preliminary results suggest that visual instruction tuning, a prevailing strategy for transitioning LLMs into MLLMs, unexpectedly and interestingly helps models attain both improved truthfulness and ethical alignment in the pure NLP context. For example, a visual-instruction-tuned LLaMA2 7B model surpasses the performance of the LLaMA2-chat 7B model, fine-tuned with over one million human annotations, on TruthfulQA-mc and Ethics benchmarks. Further analysis reveals that the improved alignment can be attributed to the superior instruction quality inherent to visual-text data. In releasing our code at github.com/UCSC-VLAA/Sight-Beyond-Text, we aspire to foster further exploration into the intrinsic value of visual-text synergies and, in a broader scope, multi-modal interactions in alignment research.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 38 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

31 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May

Other references · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
17 more
  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  6. 2019Natural Adversarial Examplescited by 1,408
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  8. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  9. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  10. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  11. 2020Aligning AI With Shared Human Valuescited by 723
  12. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
  13. 2019Adversarial Examples Improve Image Recognitioncited by 286
  14. 2021OOD-CV: A Benchmark for Robustness to Out-of-Distribution Shifts of Individual Nuisances in Natural Imagescited by 46
  15. 2021Maria: A Visual Experience Powered Conversational Agentcited by 19
  16. 2023Vision Learners Meet Web Image-Text Pairscited by 6
  17. 2023ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialoguecited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 1; 2026: 1.

14 papers.

Sort:

By area:

  1. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 108
  2. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  3. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuningcited by 64
  4. 2024VHELM: A Holistic Evaluation of Vision Language Modelscited by 53
  5. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  6. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 33
  7. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  8. 2024The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Modelscited by 20
  9. 2023Vision-Language Instruction Tuning: A Review and Analysiscited by 17
  10. 2024The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspectivecited by 15
  11. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  12. 2023How to Bridge the Gap between Modalities: Survey on Multimodal Large Language Modelcited by 3
2 more
  1. 2023What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Modelscited by 1
  2. 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  2. 2024Assessment of Multimodal Large Language Models in Alignment with Human Values
  3. 2024HonestLLM: Toward an Honest and Helpful Large Language Model
  4. 2023On the Performance of Multimodal Language Models
  5. 2025Better Reasoning with Less Data: Enhancing VLMs Through Unified Modality Scoring
  6. 2024Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
  7. 2025Rethinking the Text-Vision Reasoning Imbalance in MLLMs through the Lens of Training Recipes
  8. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark