# Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations

arXiv: https://arxiv.org/abs/2501.04675 · HTML: https://mlsanity.com/paper/2501.04675/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 8 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 12 of 12)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models](https://mlsanity.com/paper/2409.17146/) (2024)
- [Pixtral 12B](https://mlsanity.com/paper/2410.07073/) (2024)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering](https://mlsanity.com/paper/2212.09662/) (2022)
- [The Effects of Data Quality on Machine Learning Performance on Tabular Data](https://mlsanity.com/paper/2207.14529/) (2022)
- [Fine-tuning Smaller Language Models for Question Answering over Financial Documents](https://mlsanity.com/paper/2408.12337/) (2024)
- [From Data Quality to Model Quality: an Exploratory Study on Deep Learning](https://mlsanity.com/paper/1906.11882/) (2019)

## Built on by (most cited first, 5 of 5)

- [ChartAgent: A Multimodal Agent for Visually Grounded Reasoning in Complex Chart Question Answering](https://mlsanity.com/paper/2510.04514/) (2025)
- [TALENT: Table VQA via Augmented Language-Enhanced Natural-text Transcription](https://mlsanity.com/paper/2510.07098/) (2025)
- [FinMR: A Knowledge-Intensive Multimodal Benchmark for Advanced Financial Reasoning](https://mlsanity.com/paper/2510.07852/) (2025)
- [Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports](https://mlsanity.com/paper/2508.05669/) (2025)
- [A Multistage Extraction Pipeline for Long Scanned Financial Documents: An Empirical Study in Industrial KYC Workflows](https://mlsanity.com/paper/2604.26462/) (2026)

## Abstract (arXiv)

Chart interpretation is crucial for visual data analysis, but accurately extracting information from charts poses significant challenges for automated models. This study investigates the fine-tuning of DEPLOT, a modality conversion module that translates the image of a plot or chart to a linearized table, on a custom dataset of 50,000 bar charts. The dataset comprises simple, stacked, and grouped bar charts, targeting the unique structural features of these visualizations. The finetuned DEPLOT model is evaluated against its base version using a test set of 1,000 images and two metrics: Relative Mapping Similarity (RMS), which measures categorical mapping accuracy, and Relative Number Set Similarity (RNSS), which evaluates numerical interpretation accuracy. To further explore the reasoning capabilities of large language models (LLMs), we curate an additional set of 100 bar chart images paired with question answer sets. Our findings demonstrate that providing a structured intermediate table alongside the image significantly enhances LLM reasoning performance compared to direct image queries.