Multimodal Financial Foundation Models (MFFMs): Progress, Prospects, and Challenges

Financial Large Language Models (FinLLMs), such as open FinGPT and proprietary BloombergGPT, have demonstrated great potential in select areas of financial services. Beyond this earlier language-centric approach, Multimodal Financial Foundation Models (MFFMs) can digest interleaved multimodal financial data, including fundamental data, market data, data analytics, macroeconomic, and alternative data (e.g., natural language, audio, images, and video). In this position paper, presented at the MFFM Workshop joined with ACM International Conference on AI in Finance (ICAIF) 2024, we describe the progress, prospects, and challenges of MFFMs. This paper also highlights ongoing research on FinAgents in the SecureFinAI Labfootnotehttps://openfin.engineering.columbia.edu/ at Columbia University. We believe that MFFMs will enable a deeper understanding of the underlying complexity associated with numerous financial tasks and data, streamlining the operation of financial services and investment processes. Github Repo https://github.com/Open-Finance-Lab/Awesome-MFFMs/.