# Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications

arXiv: https://arxiv.org/abs/2408.11878 · HTML: https://mlsanity.com/paper/2408.11878/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 28 of its 61 references that are in the map.
- First posted on arXiv 20 Aug 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 28 of 28)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models](https://mlsanity.com/paper/2402.11684/) (2024)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [Continual Learning for Large Language Models: A Survey](https://mlsanity.com/paper/2402.01364/) (2024)
- [To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning](https://mlsanity.com/paper/2311.07574/) (2023)
- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcripts](https://mlsanity.com/paper/2210.12467/) (2022)
- [The Wall Street Neophyte: A Zero-Shot Analysis of ChatGPT Over MultiModal Stock Movement Prediction Challenges](https://mlsanity.com/paper/2304.05351/) (2023)
- [DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documents](https://mlsanity.com/paper/2311.09805/) (2023)
- [Generating Plausible Counterfactual Explanations for Deep Transformers in Financial Text Classification](https://mlsanity.com/paper/2010.12512/) (2020)
- [FinanceMath: Knowledge-Intensive Math Reasoning in Finance Domains](https://mlsanity.com/paper/2311.09797/) (2023)
- [Trade the Event: Corporate Events Detection for News-Based Event-Driven Trading](https://mlsanity.com/paper/2105.12825/) (2021)
- [FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models](https://mlsanity.com/paper/2402.10986/) (2024)
- [Financial Document Causality Detection Shared Task (FinCausal 2020)](https://mlsanity.com/paper/2012.02505/) (2020)
- [Fin-Fact: A Benchmark Dataset for Multimodal Financial Fact Checking and Explanation Generation](https://mlsanity.com/paper/2309.08793/) (2023)
- [Empowering Many, Biasing a Few: Generalist Credit Scoring through Large Language Models](https://mlsanity.com/paper/2310.00566/) (2023)
- [FiNER-ORD: Financial Named Entity Recognition Open Research Dataset](https://mlsanity.com/paper/2302.11157/) (2023)
- [FinGPT-HPC: Efficient Pretraining and Finetuning Large Language Models for Financial Applications with High-Performance Computing](https://mlsanity.com/paper/2402.13533/) (2024)

## Built on by (most cited first, 30 of 36)

- [Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2503.16252/) (2025)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [Fino1: On the Transferability of Reasoning-Enhanced LLMs and Reinforcement Learning to Finance](https://mlsanity.com/paper/2502.08127/) (2025)
- [MarS: a Financial Market Simulation Engine Powered by Generative Foundation Model](https://mlsanity.com/paper/2409.07486/) (2024)
- [Demystifying Domain-adaptive Post-training for Financial LLMs](https://mlsanity.com/paper/2501.04961/) (2025)
- [FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documents](https://mlsanity.com/paper/2411.05764/) (2024)
- [Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs](https://mlsanity.com/paper/2505.19678/) (2025)
- [Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Finance](https://mlsanity.com/paper/2410.01109/) (2024)
- [Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink](https://mlsanity.com/paper/2501.15269/) (2025)
- [Retrieval-augmented Large Language Models for Financial Time Series Forecasting](https://mlsanity.com/paper/2502.05878/) (2025)
- [FinGAIA: A Chinese Benchmark for AI Agents in Real-World Financial Domain](https://mlsanity.com/paper/2507.17186/) (2025)
- [FT-Dojo: Towards Autonomous LLM Fine-Tuning with Language Agents](https://mlsanity.com/paper/2603.01712/) (2026)
- [FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning](https://mlsanity.com/paper/2412.12567/) (2024)
- [SECQUE: A Benchmark for Evaluating Real-World Financial Analysis Capabilities](https://mlsanity.com/paper/2504.04596/) (2025)
- [Interpretable LLMs for Credit Risk: A Systematic Review and Taxonomy](https://mlsanity.com/paper/2506.04290/) (2025)
- [ReGLA: Refining Gated Linear Attention](https://mlsanity.com/paper/2502.01578/) (2025)
- [Unveiling the Landscape of LLM Deployment in the Wild: An Empirical Study](https://mlsanity.com/paper/2505.02502/) (2025)
- [Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges](https://mlsanity.com/paper/2507.18577/) (2025)
- [UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models](https://mlsanity.com/paper/2410.14059/) (2024)
- [FinAudio: A Benchmark for Audio Large Language Models in Financial Applications](https://mlsanity.com/paper/2503.20990/) (2025)
- [Truth Neurons](https://mlsanity.com/paper/2505.12182/) (2025)
- [Exploring Large Language Models for Financial Applications: Techniques, Performance, and Challenges with FinMA](https://mlsanity.com/paper/2510.05151/) (2025)
- [Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Prediction](https://mlsanity.com/paper/2510.15691/) (2025)
- [Customized FinGPT Search Agents Using Foundation Models](https://mlsanity.com/paper/2410.15284/) (2024)
- [TabReason: A Reinforcement Learning-Enhanced Reasoning LLM for Explainable Tabular Data Prediction](https://mlsanity.com/paper/2505.21807/) (2025)
- [RKEFino1: A Regulation Knowledge-Enhanced Large Language Model](https://mlsanity.com/paper/2506.05700/) (2025)
- [Fin-Ally: Pioneering the Development of an Advanced, Commonsense-Embedded Conversational AI for Money Matters](https://mlsanity.com/paper/2509.24342/) (2025)
- [Evaluating Large Language Models for Stance Detection on Financial Targets from SEC Filing Reports and Earnings Call Transcripts](https://mlsanity.com/paper/2510.23464/) (2025)
- [Market-Bench: Evaluating Large Language Models on Introductory Quantitative Trading and Market Dynamics](https://mlsanity.com/paper/2512.12264/) (2025)
- [FinAcumen: Financial Multimodal Reasoning via Self-Evolving Experience Memory Harness](https://mlsanity.com/paper/2606.17642/) (2026)

## Abstract (arXiv)

Financial LLMs hold promise for advancing financial tasks and domain-specific applications. However, they are limited by scarce corpora, weak multimodal capabilities, and narrow evaluations, making them less suited for real-world application. To address this, we introduce textitOpen-FinLLMs, the first open-source multimodal financial LLMs designed to handle diverse tasks across text, tabular, time-series, and chart data, excelling in zero-shot, few-shot, and fine-tuning settings. The suite includes FinLLaMA, pre-trained on a comprehensive 52-billion-token corpus; FinLLaMA-Instruct, fine-tuned with 573K financial instructions; and FinLLaVA, enhanced with 1.43M multimodal tuning pairs for strong cross-modal reasoning. We comprehensively evaluate Open-FinLLMs across 14 financial tasks, 30 datasets, and 4 multimodal tasks in zero-shot, few-shot, and supervised fine-tuning settings, introducing two new multimodal evaluation datasets. Our results show that Open-FinLLMs outperforms afvanced financial and general LLMs such as GPT-4, across financial NLP, decision-making, and multi-modal tasks, highlighting their potential to tackle real-world challenges. To foster innovation and collaboration across academia and industry, we release all codes (https://anonymous.4open.science/r/PIXIU2-0D70/B1D7/LICENSE) and models under OSI-approved licenses.