mlsanityData to 7 Oct 2026

Financial Instruction Following Evaluation (FIFE)

arXiv 2512.08965 ↗First posted 1 Dec 2025cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Language Models (LMs) struggle with complex, interdependent instructions, particularly in high-stakes domains like finance where precision is critical. We introduce FIFE, a novel, high-difficulty benchmark designed to assess LM instruction-following capabilities for financial analysis tasks. FIFE comprises 88 human-authored prompts and employs a verification system with chainable, verifiable constraints for fine-grained reward signals. We evaluate 53 models (proprietary, open-weight, open-source) in a zero-shot setting. Our key findings reveal a clear performance hierarchy: the top open-weight model (76.1 strict / 79.5 loose) surpasses the leading proprietary system (65.9 strict / 70.5 loose), while the best open-source models lag significantly (45.5 strict / 48.9 loose). However, even top-performing models struggle with FIFE's complex requirements, failing to achieve perfect compliance. We release our dataset and code as an open-source resource to promote research in Reinforcement Learning for the financial domain.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 49 references that are in the map. First posted on arXiv 1 Dec 2025.

Builds on

44 of 49 references.

References · 44

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  6. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  9. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 217
  10. 2025Generalizing Verifiable Instruction Followingcited by 178
  11. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 147
  12. 2022WHEN FLUE MEETS FLANG: Benchmarks and Large Pre-trained Language Model for Financial Domaincited by 117
32 more
  1. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 100
  2. 2024Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Modelscited by 99
  3. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 94
  4. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 89
  5. 2023Trillion Dollar Words: A New Financial Dataset, Task & Market Analysiscited by 73
  6. 2023INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Modelscited by 62
  7. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 62
  8. 2024DocFinQA: A Long-Context Financial Reasoning Datasetcited by 59
  9. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  10. 2023Evaluating Instruction-Tuned Large Language Models on Code Comprehension and Generationcited by 49
  11. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 43
  12. 2025Recent Advances in Large Langauge Model Benchmarks against Data Contamination: From Static to Dynamic Evaluationcited by 38
  13. 2024Open-FinLLMs: Open Multimodal Large Language Models for Financial Applicationscited by 36
  14. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  15. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  16. 2023Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructionscited by 24
  17. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  18. 2025BizFinBench: A Business-Driven Real-World Financial Benchmark for Evaluating LLMscited by 22
  19. 2024The SIFo Benchmark: Investigating the Sequential Instruction Following Ability of Large Language Modelscited by 18
  20. 2025Spotlight Your Instructions: Instruction-following with Dynamic Attention Steeringcited by 18
  21. 2025Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Modelscited by 12
  22. 2025LLMs can be easily Confused by Instructional Distractionscited by 12
  23. 2025Order Matters: Investigate the Position Bias in Multi-constraint Instruction Followingcited by 12
  24. 2025EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Modelscited by 9
  25. 2024SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysiscited by 8
  26. 2025Instruction-Tuning Data Synthesis from Scratch via Web Reconstructioncited by 8
  27. 2025Expect the Unexpected: FailSafe Long Context QA for Financecited by 7
  28. 2025DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generationcited by 7
  29. 2024FinLLMs: A Framework for Financial Reasoning Dataset Generation with Large Language Modelscited by 6
  30. 2025PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelinescited by 5
  31. 2024Golden Touchstone: A Comprehensive Bilingual Benchmark for Evaluating Financial Large Language Modelscited by 4
  32. 2025Finance Language Model Evaluation (FLaME)cited by 4

Built on by

Not cited yet.