mlsanityData to 14 Oct 2026

Fine-Tuning Vision-Language Models for Multimodal Polymer Property Prediction

arXiv 2511.05577 ↗First posted 4 Nov 2025cs.LG cond-mat.mtrl-sci cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

Abstract (arXiv)

Vision-Language Models (VLMs) have shown strong performance in tasks like visual question answering and multimodal text generation, but their effectiveness in scientific domains such as materials science remains limited. While some machine learning methods have addressed specific challenges in this field, there is still a lack of foundation models designed for broad tasks like polymer property prediction using multimodal data. In this work, we present a multimodal polymer dataset to fine-tune VLMs through instruction-tuning pairs and assess the impact of multimodality on prediction performance. Our fine-tuned models, using LoRA, outperform unimodal and baseline approaches, demonstrating the benefits of multimodal learning. Additionally, this approach reduces the need to train separate models for different properties, lowering deployment and maintenance costs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 43 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

26 of 43 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2023GPT-4 Technical Reportcited by 18,396
  6. 2024The Llama 3 Herd of Modelscited by 14,715
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2023Visual Instruction Tuningcited by 7,869
  10. 2025Qwen2.5-VL Technical Reportcited by 5,587
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
14 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  5. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  6. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  7. 2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 357
  8. 2023VeRA: Vector-based Random Matrix Adaptationcited by 300
  9. 2022SELFIES and the future of molecular string representationscited by 76
  10. 2022polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informaticscited by 48
  11. 2017Chemception: A Deep Neural Network with Minimal Chemistry Knowledge Matches the Performance of Expert-developed QSAR/QSPR Modelscited by 19
  12. 2023SCITUNE: Aligning Large Language Models with Human-Curated Scientific Multimodal Instructionscited by 19
  13. 2025Multimodal machine learning with large language embedding model for polymer property predictioncited by 5
  14. 2025Benchmarking Large Language Models for Polymer Property Predictionscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Large language models, physics-based modeling, experimental measurements: the trinity of data-scarce learning of polymer properties
  2. 2026PolyReal: A Benchmark for Real-World Polymer Science Workflows
  3. 2026Teaching and Evaluating LLMs to Reason About Polymer Design Related Tasks
  4. 2025MolVision: Molecular Property Prediction with Vision Language Models
  5. 2026Polymer-Agent: Large Language Model Agent for Polymer Design
  6. 2025MatQnA: A Benchmark Dataset for Multi-modal Large Language Models in Materials Characterization and Analysis
  7. 2025Can Multimodal LLMs See Materials Clearly? A Multimodal Benchmark on Materials Characterization
  8. 2024Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions