mlsanityData to 7 Oct 2026

VSLLaVA: a pipeline of large multimodal foundation model for industrial vibration signal analysis

arXiv 2409.07482 ↗First posted 3 Sep 2024eess.SP cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

While Large Multimodal Models (LMMs) excel in general multimodal tasks, they lack the domain-specific knowledge for industrial vibration signal analysis. This paper introduces VSLLaVA, a comprehensive pipeline that utilizes expert knowledge-guided instruction tuning and evaluation to create an end-to-end LMM for signal analysis. To achieve this, we construct a novel Signal-Question-Answer (SQA) dataset using an expert rule-based signal generator. This dataset facilitates a two-stage learning procedure. The first step is efficient instruction fine-tuning with Low-Rank Adaptation (LoRA), which imparts specialized signal identification capabilities. Subsequently, we designed a tailored Group Relative Policy Optimization (GRPO) to refine the reasoning capabilities and enhance classification robustness. Then, a dual-mode evaluation framework is proposed, combining an LLM referee with expert rules for semantic assessment using quantitative metrics for numerical and textual accuracy, which reveals that VSLLaVA significantly improves performance in signal type identification and parameter analysis, and makes progress in the identification and parameter analysis of fault-related signals. This research demonstrates a viable approach for developing specialized foundational models for complex industrial applications and marks a transition from conventional task-specific systems to a cohesive, interactive foundational model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 77 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

29 of 77 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  10. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  11. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  12. 2023Vision-Language Models for Vision Tasks: A Surveycited by 834
17 more
  1. 2020A Survey on Visual Transformercited by 653
  2. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  3. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  4. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 272
  5. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  6. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  7. 2023Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Studycited by 85
  8. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  9. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  10. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 56
  11. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  12. 2024A Review of Multi-Modal Large Language and Vision Modelscited by 30
  13. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 28
  14. 2020Machine Learning for Reliability Engineering and Safety Applications: Review of Current Status and Future Opportunitiescited by 22
  15. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  16. 2024BearLLM: A Prior Knowledge-Enhanced Bearing Health Management Framework with Unified Vibration Signal Representationcited by 4
  17. 2023Survey on Foundation Models for Prognostics and Health Management in Industrial Cyber-Physical Systemscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Labels Are Scarce: An Oscillatory State Space Model for Vibration Diagnosis
  2. 2026VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence
  3. 2025Towards Open-Vocabulary Industrial Defect Understanding with a Large-Scale Multimodal Dataset
  4. 2025Prescriptive Agents based on RAG for Automated Maintenance (PARAM)
  5. 2026Lightweight Multimodal LLM-Enabled Cost-Effective Defect Grading of Power Transmission Equipment
  6. 2025SoM-1K: A Thousand-Problem Benchmark Dataset for Strength of Materials
  7. 2025Can Multimodal Large Language Models be Guided to Improve Industrial Anomaly Detection?
  8. 2024EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark