mlsanityData to 14 Oct 2026

SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructions

arXiv 2307.01139 ↗First posted 3 Jul 2023cs.CV cs.AI cs.CLcited by 19

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

Abstract (arXiv)

Instruction finetuning is a popular paradigm to align large language models (LLM) with human intent. Despite its popularity, this idea is less explored in improving LLMs to align existing foundation models with scientific disciplines, concepts and goals. In this work, we present textitSciTune as a tuning framework to improve the ability of LLMs to follow multimodal instructions generated from scientific publications. To test our methodology, we train a large multimodal model LLaMA-SciTune that connects a vision encoder and LLM for science-focused visual and language understanding. LLaMA-SciTune significantly outperforms the state-of-the-art models in the generated figure types and captions in SciCap and VisText benchmarks. In comparison to the models that are finetuned with synthetic data only, LLaMA-SciTune surpasses human performance on average and in many sub-categories on the ScienceQA benchmark. Our results demonstrate that human-generated scientific multimodal instructions remain highly valuable in tuning LLMs to perform well on science tasks, despite their lower volume and relative scarcity compared to synthetic data. We publicly release the SciTune codebase https://github.com/pnnl/scitune.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 14 of its 38 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

14 of 38 references.

References · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  5. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  6. 2021Ethical and social risks of harm from Language Modelscited by 966
  7. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 781
  8. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  9. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 274
  10. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  11. 2023VisText: A Benchmark for Semantically Rich Chart Captioningcited by 112
  12. 2022Mind's Eye: Grounded Language Model Reasoning through Simulationcited by 84
2 more
  1. 2021SciCap: Generating Captions for Scientific Figurescited by 77
  2. 2021Mitigating Political Bias in Language Models Through Reinforced Calibrationcited by 73

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 8; 2025: 4; 2026: 1.

19 papers.

Sort:

By area:

  1. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  2. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 137
  3. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 79
  4. 2024KAM-CoT: Knowledge Augmented Multimodal Chain-of-Thoughts Reasoningcited by 60
  5. 2024Cantor: Inspiring Multimodal Chain-of-Thought of MLLMcited by 34
  6. 2025HM-RAG: Hierarchical Multi-Agent Multimodal Retrieval Augmented Generationcited by 31
  7. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  8. 2023Thinking Like an Expert:Multimodal Hypergraph-of-Thought (HoT) Reasoning to boost Foundation Modalscited by 24
  9. 2023Boosting the Power of Small Multimodal Reasoning Models to Match Larger Models with Self-Consistency Trainingcited by 22
  10. 2023Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Modelscited by 18
  11. 2024A Picture Is Worth a Graph: A Blueprint Debate Paradigm for Multimodal Reasoningcited by 11
  12. 2024Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reportscited by 6
7 more
  1. 2024SciCapenter: Supporting Caption Composition for Scientific Figures with Machine-Generated Captions and Ratingscited by 5
  2. 2024Faithful Chart Summarization with ChaTS-Picited by 5
  3. 2024Generalist Multimodal AI: A Review of Architectures, Challenges and Opportunitiescited by 3
  4. 2025NL2GenSym: Natural Language to Generative Symbolic Rules for SOAR Cognitive Architecture via Large Language Modelscited by 1
  5. 2025Cross-Disciplinary Knowledge Retrieval and Synthesis: A Compound AI Architecture for Scientific Discoverycited by 1
  6. 2025Fine-Tuning Vision-Language Models for Multimodal Polymer Property Predictioncited by 0
  7. 2026HANIA: Planner-Guided Multimodal Graph Evidence Selection for Grounded Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MAC: A Live Benchmark for Multimodal Large Language Models in Scientific Understanding
  2. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?
  3. 2023Visual Instruction Tuning
  4. 2023StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data
  5. 2023Large Language Models as Automated Aligners for benchmarking Vision-Language Models
  6. 2025LLaVA-MORE: A Comparative Study of LLMs and Visual Backbones for Enhanced Visual Instruction Tuning
  7. 2024MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment
  8. 2026Innovator-VL: A Multimodal Large Language Model for Scientific Discovery