# FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysis

arXiv: https://arxiv.org/abs/2308.01430 · HTML: https://mlsanity.com/paper/2308.01430/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 10 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [Language Is Not All You Need: Aligning Perception with Language Models](https://mlsanity.com/paper/2302.14045/) (2023)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)

## Built on by (most cited first, 28 of 28)

- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning](https://mlsanity.com/paper/2505.11049/) (2025)
- [Visual Large Language Models for Generalized and Specialized Applications](https://mlsanity.com/paper/2501.02765/) (2025)
- [FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models](https://mlsanity.com/paper/2402.10986/) (2024)
- [MLLM-CL: Continual Learning for Multimodal Large Language Models](https://mlsanity.com/paper/2506.05453/) (2025)
- [MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark](https://mlsanity.com/paper/2508.07307/) (2025)
- [No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarks](https://mlsanity.com/paper/2403.06249/) (2024)
- [Agentic Trading: When LLM Agents Meet Financial Markets](https://mlsanity.com/paper/2605.19337/) (2026)
- [Co-audit: tools to help humans double-check AI-generated content](https://mlsanity.com/paper/2310.01297/) (2023)
- [FinTeam: A Multi-Agent Collaborative Intelligence System for Comprehensive Financial Scenarios](https://mlsanity.com/paper/2507.10448/) (2025)
- [Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation](https://mlsanity.com/paper/2602.00722/) (2026)
- [Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Models](https://mlsanity.com/paper/2411.06272/) (2024)
- [Bridging Language Models and Financial Analysis](https://mlsanity.com/paper/2503.22693/) (2025)
- [Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges](https://mlsanity.com/paper/2507.18577/) (2025)
- [Insight Miner: A Time Series Analysis Dataset for Cross-Domain Alignment with Natural Language](https://mlsanity.com/paper/2512.11251/) (2025)
- [FinLMM-R1: Enhancing Financial Reasoning in LMM through Scalable Data and Reward Design](https://mlsanity.com/paper/2506.13066/) (2025)
- [FediLoRA: Practical Federated Fine-Tuning of Foundation Models Under Missing-Modality Constraints](https://mlsanity.com/paper/2509.06984/) (2025)
- [Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement](https://mlsanity.com/paper/2602.10138/) (2026)
- [Echoes of ownership: Adversarial-guided dual injection for copyright protection in MLLMs](https://mlsanity.com/paper/2602.18845/) (2026)
- [DBOT: Artificial Intelligence for Systematic Long-Term Investing](https://mlsanity.com/paper/2504.05639/) (2025)
- [BlindSight: Harnessing Sparsity for Efficient Vision-Language Models](https://mlsanity.com/paper/2507.09071/) (2025)
- [PyFi: Toward Pyramid-like Financial Image Understanding for VLMs via Adversarial Agents](https://mlsanity.com/paper/2512.14735/) (2025)
- [UniFinEval: Towards Unified Evaluation of Financial Multimodal Models across Text, Images and Videos](https://mlsanity.com/paper/2601.22162/) (2026)
- [MAny: Merge Anything for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2604.14016/) (2026)
- [KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation](https://mlsanity.com/paper/2606.16256/) (2026)
- [FinSAgent: Corpus-Aligned Multi-Agent RAG Framework for Evidence-Grounded SEC Filing Question Answering](https://mlsanity.com/paper/2607.18102/) (2026)
- [SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2607.20511/) (2026)
- [Progressive Multimodal Alignment for Continual Instruction Tuning](https://mlsanity.com/paper/2607.26947/) (2026)

## Abstract (arXiv)

In this paper, we propose FinVis-GPT, a novel multimodal large language model (LLM) specifically designed for financial chart analysis. By leveraging the power of LLMs and incorporating instruction tuning and multimodal capabilities, FinVis-GPT is capable of interpreting financial charts and providing valuable analysis. To train FinVis-GPT, a financial task oriented dataset was generated for pre-training alignment and instruction tuning, comprising various types of financial charts and their corresponding descriptions. We evaluate the model performance via several case studies due to the time limit, and the promising results demonstrated that FinVis-GPT is superior in various financial chart related tasks, including generating descriptions, answering questions and predicting future market trends, surpassing existing state-of-the-art multimodal LLMs. The proposed FinVis-GPT serves as a pioneering effort in utilizing multimodal LLMs in the finance domain and our generated dataset will be release for public use in the near future to speedup related research.