# Dynamic Datasets and Market Environments for Financial Reinforcement Learning

arXiv: https://arxiv.org/abs/2304.13174 · HTML: https://mlsanity.com/paper/2304.13174/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 113 references that are in the map.
- First posted on arXiv 25 Apr 2023.

## Builds on (most cited first, 30 of 41)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Tune: A Research Platform for Distributed Model Selection and Training](https://mlsanity.com/paper/1807.05118/) (2018)
- [FinBERT: Financial Sentiment Analysis with Pre-trained Language Models](https://mlsanity.com/paper/1908.10063/) (2019)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Data-centric Artificial Intelligence: A Survey](https://mlsanity.com/paper/2303.10158/) (2023)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Embodied Intelligence via Learning and Evolution](https://mlsanity.com/paper/2102.02202/) (2021)
- [Data Collection and Quality Challenges in Deep Learning: A Data-Centric AI Perspective](https://mlsanity.com/paper/2112.06409/) (2021)
- [Recent Advances in Reinforcement Learning in Finance](https://mlsanity.com/paper/2112.04553/) (2021)
- [DataPerf: Benchmarks for Data-Centric AI Development](https://mlsanity.com/paper/2207.10062/) (2022)
- [FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance](https://mlsanity.com/paper/2011.09607/) (2020)
- [Deep Reinforcement Learning for Trading](https://mlsanity.com/paper/1911.10107/) (2019)
- [NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning](https://mlsanity.com/paper/2102.00714/) (2021)
- [Differentially Private Secure Multi-Party Computation for Federated Learning in Financial Applications](https://mlsanity.com/paper/2010.05867/) (2020)
- [FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Finance](https://mlsanity.com/paper/2111.09395/) (2021)
- [Practical Deep Reinforcement Learning Approach for Stock Trading](https://mlsanity.com/paper/1811.07522/) (2018)
- [Data-centric AI: Perspectives and Challenges](https://mlsanity.com/paper/2301.04819/) (2023)
- [ABIDES-Gym: Gym Environments for Multi-Agent Discrete Event Simulation and Application to Financial Markets](https://mlsanity.com/paper/2110.14771/) (2021)
- [A Benchmark Environment Motivated by Industrial Control Problems](https://mlsanity.com/paper/1709.09480/) (2017)
- [Towards Realistic Market Simulations: a Generative Adversarial Networks Approach](https://mlsanity.com/paper/2110.13287/) (2021)

## Built on by (most cited first, 15 of 15)

- [FinBen: A Holistic Financial Benchmark for Large Language Models](https://mlsanity.com/paper/2402.12659/) (2024)
- [FinGPT: Democratizing Internet-scale Data for Financial Large Language Models](https://mlsanity.com/paper/2307.10485/) (2023)
- [Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning](https://mlsanity.com/paper/2312.17493/) (2023)
- [FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models](https://mlsanity.com/paper/2405.14767/) (2024)
- [Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?](https://mlsanity.com/paper/2505.07078/) (2025)
- [FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.15800/) (2025)
- [MUSO: Achieving Exact Machine Unlearning in Over-Parameterized Regimes](https://mlsanity.com/paper/2410.08557/) (2024)
- [Reinforcement-Learning Portfolio Allocation with Dynamic Embedding of Market Information](https://mlsanity.com/paper/2501.17992/) (2025)
- [LLMs Meet Finance: Fine-Tuning Foundation Models for the Open FinLLM Leaderboard](https://mlsanity.com/paper/2504.13125/) (2025)
- [Hierarchical Reinforced Trader (HRT): A Bi-Level Approach for Optimizing Stock Selection and Execution](https://mlsanity.com/paper/2410.14927/) (2024)
- [Customized FinGPT Search Agents Using Foundation Models](https://mlsanity.com/paper/2410.15284/) (2024)
- [Janus-Q: End-to-End Event-Driven Trading via Hierarchical-Gated Reward Modeling](https://mlsanity.com/paper/2602.19919/) (2026)
- [Revisiting Ensemble Methods for Stock Trading and Crypto Trading Tasks at ACM ICAIF FinRL Contest 2023-2024](https://mlsanity.com/paper/2501.10709/) (2025)
- [Directly Learning Stock Trading Strategies Through Profit Guided Loss Functions](https://mlsanity.com/paper/2507.19639/) (2025)
- [Compositional Behavioral Semantics for State Abstraction in Reinforcement Learning](https://mlsanity.com/paper/2606.25357/) (2026)

## Abstract (arXiv)

The financial market is a particularly challenging playground for deep reinforcement learning due to its unique feature of dynamic datasets. Building high-quality market environments for training financial reinforcement learning (FinRL) agents is difficult due to major factors such as the low signal-to-noise ratio of financial data, survivorship bias of historical data, and model overfitting. In this paper, we present FinRL-Meta, a data-centric and openly accessible library that processes dynamic datasets from real-world markets into gym-style market environments and has been actively maintained by the AI4Finance community. First, following a DataOps paradigm, we provide hundreds of market environments through an automatic data curation pipeline. Second, we provide homegrown examples and reproduce popular research papers as stepping stones for users to design new trading strategies. We also deploy the library on cloud platforms so that users can visualize their own results and assess the relative performance via community-wise competitions. Third, we provide dozens of Jupyter/Python demos organized into a curriculum and a documentation website to serve the rapidly growing community. The open-source codes for the data curation pipeline are available at https://github.com/AI4Finance-Foundation/FinRL-Meta