# UCFE: A User-Centric Financial Expertise Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2410.14059 · HTML: https://mlsanity.com/paper/2410.14059/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 62 references that are in the map.
- First posted on arXiv 17 Oct 2024.

## Builds on (most cited first, 21 of 21)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark](https://mlsanity.com/paper/2409.02813/) (2024)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance](https://mlsanity.com/paper/2306.05443/) (2023)
- [EconAgent: Large Language Model-Empowered Agents for Simulating Macroeconomic Activities](https://mlsanity.com/paper/2310.10436/) (2023)
- [Observational Scaling Laws and the Predictability of Language Model Performance](https://mlsanity.com/paper/2405.10938/) (2024)
- [XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters](https://mlsanity.com/paper/2305.12002/) (2023)
- [A Survey of Large Language Models for Financial Applications: Progress, Prospects and Challenges](https://mlsanity.com/paper/2406.11903/) (2024)
- [Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates](https://mlsanity.com/paper/2408.13006/) (2024)
- [Are ChatGPT and GPT-4 General-Purpose Solvers for Financial Text Analytics? A Study on Several Typical Tasks](https://mlsanity.com/paper/2305.05862/) (2023)
- [Open-FinLLMs: Open Multimodal Large Language Models for Financial Applications](https://mlsanity.com/paper/2408.11878/) (2024)
- [Financial Statement Analysis with Large Language Models](https://mlsanity.com/paper/2407.17866/) (2024)
- [Leveraging Vision-Language Models for Granular Market Change Prediction](https://mlsanity.com/paper/2301.10166/) (2023)
- [EUCA: the End-User-Centered Explainable AI Framework](https://mlsanity.com/paper/2102.02437/) (2021)
- [Can LLMs Learn Macroeconomic Narratives from Social Media?](https://mlsanity.com/paper/2406.12109/) (2024)
- [Can Large Language Models be Used to Provide Psychological Counselling? An Analysis of GPT-4-Generated Responses Using Role-play Dialogues](https://mlsanity.com/paper/2402.12738/) (2024)
- [Generative AI for Business Strategy: Using Foundation Models to Create Business Strategy Tools](https://mlsanity.com/paper/2308.14182/) (2023)

## Built on by (most cited first, 2 of 2)

- [AlphaForgeBench: Benchmarking End-to-End Trading Strategy Design with Large Language Models](https://mlsanity.com/paper/2602.18481/) (2026)
- [IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking](https://mlsanity.com/paper/2609.29167/) (2026)

## Abstract (arXiv)

This paper introduces the UCFE: User-Centric Financial Expertise benchmark, an innovative framework designed to evaluate the ability of large language models (LLMs) to handle complex real-world financial tasks. UCFE benchmark adopts a hybrid approach that combines human expert evaluations with dynamic, task-specific interactions to simulate the complexities of evolving financial scenarios. Firstly, we conducted a user study involving 804 participants, collecting their feedback on financial tasks. Secondly, based on this feedback, we created our dataset that encompasses a wide range of user intents and interactions. This dataset serves as the foundation for benchmarking 11 LLMs services using the LLM-as-Judge methodology. Our results show a significant alignment between benchmark scores and human preferences, with a Pearson correlation coefficient of 0.78, confirming the effectiveness of the UCFE dataset and our evaluation approach. UCFE benchmark not only reveals the potential of LLMs in the financial domain but also provides a robust framework for assessing their performance and user satisfaction.