# Behavioral Bias of Vision-Language Models: A Behavioral Finance View

arXiv: https://arxiv.org/abs/2409.15256 · HTML: https://mlsanity.com/paper/2409.15256/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 13 of 13)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models](https://mlsanity.com/paper/2403.18814/) (2024)
- [MobileVLM V2: Faster and Stronger Baseline for Vision Language Model](https://mlsanity.com/paper/2402.03766/) (2024)
- [InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning](https://mlsanity.com/paper/2309.13064/) (2023)
- [A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law](https://mlsanity.com/paper/2405.01769/) (2024)
- [Financial Statement Analysis with Large Language Models](https://mlsanity.com/paper/2407.17866/) (2024)

## Built on by (most cited first, 3 of 3)

- [AI Agent Behavioral Science](https://mlsanity.com/paper/2506.06366/) (2025)
- [A Systematic Analysis of the Impact of Persona Steering on LLM Capabilities](https://mlsanity.com/paper/2604.11048/) (2026)
- [InvestAlign: Overcoming Data Scarcity in Aligning Large Language Models with Investor Decision-Making Processes under Herd Behavior](https://mlsanity.com/paper/2507.06528/) (2025)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) evolve rapidly as Large Language Models (LLMs) was equipped with vision modules to create more human-like models. However, we should carefully evaluate their applications in different domains, as they may possess undesired biases. Our work studies the potential behavioral biases of LVLMs from a behavioral finance perspective, an interdisciplinary subject that jointly considers finance and psychology. We propose an end-to-end framework, from data collection to new evaluation metrics, to assess LVLMs' reasoning capabilities and the dynamic behaviors manifested in two established human financial behavioral biases: recency bias and authority bias. Our evaluations find that recent open-source LVLMs such as LLaVA-NeXT, MobileVLM-V2, Mini-Gemini, MiniCPM-Llama3-V 2.5 and Phi-3-vision-128k suffer significantly from these two biases, while the proprietary model GPT-4o is negligibly impacted. Our observations highlight directions in which open-source models can improve. The code is available at https://github.com/mydcxiao/vlm_behavioral_fin.