mlsanityData to 7 Oct 2026

FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

arXiv 2402.10986 ↗First posted 16 Feb 2024cs.CL cs.AIcited by 32

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

also in 1 more
Abstract (arXiv)

We introduce FinTral, a suite of state-of-the-art multimodal large language models (LLMs) built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We enhance FinTral with domain-specific pretraining, instruction fine-tuning, and RLAIF training by exploiting a large collection of textual and visual datasets we curate for this work. We also introduce an extensive benchmark featuring nine tasks and 25 datasets for evaluation, including hallucinations in the financial domain. Our FinTral model trained with direct preference optimization employing advanced Tools and Retrieval methods, dubbed FinTral-DPO-T&R, demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine tasks, marking a significant advancement in AI-driven financial technology. We also demonstrate that FinTral has the potential to excel in real-time analysis and decision-making in diverse financial contexts. The GitHub repository for FinTral is available at urlhttps://github.com/UBC-NLP/fintral.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 29 of its 55 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

29 of 55 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  10. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  11. 2023BloombergGPT: A Large Language Model for Financecited by 923
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
17 more
  1. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  2. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  3. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  4. 2019FinBERT: Financial Sentiment Analysis with Pre-trained Language Modelscited by 378
  5. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  6. 2023FinanceBench: A New Benchmark for Financial Question Answeringcited by 261
  7. 2022ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answeringcited by 194
  8. 2020Impact of News on the Commodity Market: Dataset and Resultscited by 80
  9. 2023Trillion Dollar Words: A New Financial Dataset, Task & Market Analysiscited by 73
  10. 2023DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuningcited by 64
  11. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  12. 2019Doc2EDAG: An End-to-End Document-level Framework for Chinese Financial Event Extractioncited by 54
  13. 2023Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucinationcited by 41
  14. 2021Trade the Event: Corporate Events Detection for News-Based Event-Driven Tradingcited by 39
  15. 2021EDGAR-CORPUS: Billions of Tokens Make The World Go Roundcited by 32
  16. 2023FinVis-GPT: A Multimodal Large Language Model for Financial Chart Analysiscited by 27
  17. 2023FiNER-ORD: Financial Named Entity Recognition Open Research Datasetcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 23; 2026: 1.

32 papers.

Sort:

By area:

  1. 2024A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challengescited by 86
  2. 2024FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Modelscited by 39
  3. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
  4. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  5. 2024MarS: a Financial Market Simulation Engine Powered by Generative Foundation Modelcited by 27
  6. 2024Mixing It Up: The Cocktail Effect of Multi-Task Fine-Tuning on LLM Performance -- A Case Study in Financecited by 15
  7. 2025FinLoRA: Benchmarking LoRA Methods for Fine-Tuning LLMs on Financial Datasetscited by 11
  8. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 9
  9. 2024No Language is an Island: Unifying Chinese and English in Financial Large Language Models, Instruction Data, and Benchmarkscited by 7
  10. 2025STEER-ME: Assessing the Microeconomic Reasoning of Large Language Modelscited by 7
  11. 2025Reasoning or Overthinking: Evaluating Large Language Models on Financial Sentiment Analysiscited by 5
  12. 2025FinCoT: Grounding Chain-of-Thought in Expert Financial Reasoningcited by 5
20 more
  1. 2024Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Modelscited by 4
  2. 2025Bridging Language Models and Financial Analysiscited by 4
  3. 2025CFBenchmark-MM: Chinese Financial Assistant Benchmark for Multimodal Large Language Modelcited by 4
  4. 2025Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challengescited by 4
  5. 2025MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMscited by 4
  6. 2026From Feedback Loops to Policy Updates: Reinforcement Fine-Tuning for LLM-Based Alpha Factor Discoverycited by 4
  7. 2024A Dutch Financial Large Language Modelcited by 2
  8. 2025FinAudio: A Benchmark for Audio Large Language Models in Financial Applicationscited by 2
  9. 2025FinLMM-R1: Enhancing Financial Reasoning in LMM through Scalable Data and Reward Designcited by 2
  10. 2025Exploring Large Language Models for Financial Applications: Techniques, Performance, and Challenges with FinMAcited by 2
  11. 2025FinMR: A Knowledge-Intensive Multimodal Benchmark for Advanced Financial Reasoningcited by 2
  12. 2025Fin-Ally: Pioneering the Development of an Advanced, Commonsense-Embedded Conversational AI for Money Matterscited by 1
  13. 2025Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Predictioncited by 1
  14. 2025RedOne 2.0: Rethinking Domain-specific LLM Post-Training in Social Networking Servicescited by 1
  15. 2025DBOT: Artificial Intelligence for Systematic Long-Term Investingcited by 0
  16. 2025Towards Competent AI for Fundamental Analysis in Finance: A Benchmark Dataset and Evaluationcited by 0
  17. 2025From Time Series Analysis to Question Answering: A Survey in the LLM Eracited by 0
  18. 2025Large Language Models Acing Chartered Accountancycited by 0
  19. 2025TULIP: Adapting Open-Source Large Language Models for Underrepresented Languages and Specialized Financial Taskscited by 0
  20. 2025PG-CE: A Progressive Generation Dataset with Constraint Enhancement for Controllable Text Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation
  2. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learning
  3. 2024FinBen: A Holistic Financial Benchmark for Large Language Models
  4. 2025FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging
  5. 2025FinBERT2: A Specialized Bidirectional Encoder for Bridging the Gap in Finance-Specific Deployment of Large Language Models
  6. 2026Evaluating Financial Intelligence in Large Language Models: Benchmarking SuperInvesting AI with LLM Engines
  7. 2026Unlocking Data Value in Finance: A Study on Distillation and Difficulty-Aware Training
  8. 2026FinForge: Semi-Synthetic Financial Benchmark Generation