# Moira: Language-driven Hierarchical Reinforcement Learning for Pair Trading

arXiv: https://arxiv.org/abs/2605.01954 · HTML: https://mlsanity.com/paper/2605.01954/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 63 references that are in the map.
- First posted on arXiv 3 May 2026.

## Builds on (most cited first, 19 of 19)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [TradingAgents: Multi-Agents LLM Financial Trading Framework](https://mlsanity.com/paper/2412.20138/) (2024)
- [FinMem: A Performance-Enhanced LLM Trading Agent with Layered Memory and Character Design](https://mlsanity.com/paper/2311.13743/) (2023)
- [InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning](https://mlsanity.com/paper/2309.13064/) (2023)
- [Alpha-GPT: Human-AI Interactive Alpha Mining for Quantitative Investment](https://mlsanity.com/paper/2308.00016/) (2023)
- [AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework](https://mlsanity.com/paper/2403.12582/) (2024)
- [How Susceptible are LLMs to Influence in Prompts?](https://mlsanity.com/paper/2408.11865/) (2024)
- [QuantAgent: Seeking Holy Grail in Trading by Self-Improving Large Language Model](https://mlsanity.com/paper/2402.03755/) (2024)
- [Can Large Language Models Beat Wall Street? Unveiling the Potential of AI in Stock Selection](https://mlsanity.com/paper/2401.03737/) (2024)
- [FLAG-Trader: Fusion LLM-Agent with Gradient-based Reinforcement Learning for Financial Trading](https://mlsanity.com/paper/2502.11433/) (2025)
- [When Agents Trade: Live Multi-Market Trading Benchmark for LLM Agents](https://mlsanity.com/paper/2510.11695/) (2025)
- [StockGPT: A GenAI Model for Stock Prediction and Trading](https://mlsanity.com/paper/2404.05101/) (2024)
- [Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methods](https://mlsanity.com/paper/2506.01901/) (2025)
- [Reinforcement Learning Pair Trading: A Dynamic Scaling approach](https://mlsanity.com/paper/2407.16103/) (2024)

## Built on by (most cited first, 1 of 1)

- [Herculean: An Agentic Benchmark for Financial Intelligence](https://mlsanity.com/paper/2605.14355/) (2026)

## Abstract (arXiv)

Many sequential decision-making problems exhibit hierarchical structure, where high-level semantic choices constrain downstream actions and feedback is delayed and ambiguous. Learning in such settings is challenging due to credit assignment: performance degradation may arise from flawed abstractions, suboptimal execution, or their interaction. We study this challenge through pair trading, a domain that naturally combines long-horizon semantic reasoning for asset pair selection with short-horizon execution under partial observability. We formulate pair trading as a hierarchical reinforcement learning problem and propose a language-driven optimization framework in which both high-level and low-level policies are parameterized by large language models (LLMs) and optimized exclusively through prompt updates. Our approach leverages pretrained LLMs as hierarchical policies and uses trajectory- and episode-level textual feedback to adapt abstractions and execution without gradient-based fine-tuning. By explicitly separating abstraction selection from execution, the framework reduces non-stationarity across hierarchical levels and enables targeted adaptation under delayed feedback. Experiments on real-world market data show consistent improvements over traditional and LLM-based baselines, demonstrating the effectiveness of language-driven hierarchical reinforcement learning.