# Ordering-Based Causal Discovery with Reinforcement Learning

arXiv: https://arxiv.org/abs/2105.06631 · HTML: https://mlsanity.com/paper/2105.06631/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 41 references that are in the map.
- First posted on arXiv 14 May 2021.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [DAGs with NO TEARS: Continuous Optimization for Structure Learning](https://mlsanity.com/paper/1803.01422/) (2018)
- [Attention, Learn to Solve Routing Problems!](https://mlsanity.com/paper/1803.08475/) (2018)
- [Neural Combinatorial Optimization with Reinforcement Learning](https://mlsanity.com/paper/1611.09940/) (2016)
- [Learning Combinatorial Optimization Algorithms over Graphs](https://mlsanity.com/paper/1704.01665/) (2017)
- [DAG-GNN: DAG Structure Learning with Graph Neural Networks](https://mlsanity.com/paper/1904.10098/) (2019)
- [Causal Discovery with Continuous Additive Noise Models](https://mlsanity.com/paper/1309.6779/) (2013)
- [Gradient-Based Neural DAG Learning](https://mlsanity.com/paper/1906.02226/) (2019)
- [Learning Sparse Nonparametric DAGs](https://mlsanity.com/paper/1909.13189/) (2019)
- [CAM: Causal additive models, high-dimensional order search and penalized regression](https://mlsanity.com/paper/1310.1533/) (2013)
- [Causal Discovery with Reinforcement Learning](https://mlsanity.com/paper/1906.04477/) (2019)
- [Masked Gradient-Based Causal Structure Learning](https://mlsanity.com/paper/1910.08527/) (2019)
- [A Graph Autoencoder Approach to Causal Structure Learning](https://mlsanity.com/paper/1911.07420/) (2019)
- [Ordering-Based Causal Structure Learning in the Presence of Latent Variables](https://mlsanity.com/paper/1910.09014/) (2019)

## Built on by (most cited first, 30 of 55)

- [Score matching enables causal discovery of nonlinear additive noise models](https://mlsanity.com/paper/2203.04413/) (2022)
- [Learning to Induce Causal Structure](https://mlsanity.com/paper/2204.04875/) (2022)
- [Diffusion Models for Causal Discovery via Topological Ordering](https://mlsanity.com/paper/2210.06201/) (2022)
- [gCastle: A Python Toolbox for Causal Discovery](https://mlsanity.com/paper/2111.15155/) (2021)
- [A Survey on Causal Discovery Methods for I.I.D. and Time Series Data](https://mlsanity.com/paper/2303.15027/) (2023)
- [MissDAG: Causal Discovery in the Presence of Missing Data with Continuous Additive Noise Models](https://mlsanity.com/paper/2205.13869/) (2022)
- [Reinforcement learning on graphs: A survey](https://mlsanity.com/paper/2204.06127/) (2022)
- [Causal Deep Learning](https://mlsanity.com/paper/2303.02186/) (2023)
- [FedDAG: Federated DAG Structure Learning](https://mlsanity.com/paper/2112.03555/) (2021)
- [Optimizing NOTEARS Objectives via Topological Swaps](https://mlsanity.com/paper/2305.17277/) (2023)
- [Sample Complexity Bounds for Score-Matching: Causal Discovery and Generative Modeling](https://mlsanity.com/paper/2310.18123/) (2023)
- [Graph Reinforcement Learning for Combinatorial Optimization: A Survey and Unifying Perspective](https://mlsanity.com/paper/2404.06492/) (2024)
- [Causal-Copilot: An Autonomous Causal Analysis Agent](https://mlsanity.com/paper/2504.13263/) (2025)
- [Computably Continuous Reinforcement-Learning Objectives are PAC-learnable](https://mlsanity.com/paper/2303.05518/) (2023)
- [GFlowCausal: Generative Flow Networks for Causal Discovery](https://mlsanity.com/paper/2210.08185/) (2022)
- [Reinforcement Learning for Generative AI: A Survey](https://mlsanity.com/paper/2308.14328/) (2023)
- [Demystifying amortized causal discovery with transformers](https://mlsanity.com/paper/2405.16924/) (2024)
- [CIPCaD-Bench: Continuous Industrial Process datasets for benchmarking Causal Discovery methods](https://mlsanity.com/paper/2208.01529/) (2022)
- [Novel Ordering-based Approaches for Causal Structure Learning in the Presence of Unobserved Variables](https://mlsanity.com/paper/2208.06935/) (2022)
- [Comprehensive Review and Empirical Evaluation of Causal Discovery Algorithms for Numerical Data](https://mlsanity.com/paper/2407.13054/) (2024)
- [CORE: Towards Scalable and Efficient Causal Discovery with Reinforcement Learning](https://mlsanity.com/paper/2401.16974/) (2024)
- [Optimal Transport for Structure Learning Under Missing Data](https://mlsanity.com/paper/2402.15255/) (2024)
- [Order-based Structure Learning with Normalizing Flows](https://mlsanity.com/paper/2308.07480/) (2023)
- [COKE: Causal Discovery with Chronological Order and Expert Knowledge in High Proportion of Missing Manufacturing Data](https://mlsanity.com/paper/2407.12254/) (2024)
- [Shortcuts for causal discovery of nonlinear models by score matching](https://mlsanity.com/paper/2310.14246/) (2023)
- [Tree Search in DAG Space with Model-based Reinforcement Learning for Causal Discovery](https://mlsanity.com/paper/2310.13576/) (2023)
- [Reinforcement Learning for Causal Discovery without Acyclicity Constraints](https://mlsanity.com/paper/2408.13448/) (2024)
- [ACCESS : A Benchmark for Abstract Causal Event Discovery and Reasoning](https://mlsanity.com/paper/2502.08148/) (2025)
- [Policy Gradient Algorithms with Monte Carlo Tree Learning for Non-Markov Decision Processes](https://mlsanity.com/paper/2206.01011/) (2022)
- [Reframed GES with a Neural Conditional Dependence Measure](https://mlsanity.com/paper/2206.08531/) (2022)

## Abstract (arXiv)

It is a long-standing question to discover causal relations among a set of variables in many empirical sciences. Recently, Reinforcement Learning (RL) has achieved promising results in causal discovery from observational data. However, searching the space of directed graphs and enforcing acyclicity by implicit penalties tend to be inefficient and restrict the existing RL-based method to small scale problems. In this work, we propose a novel RL-based approach for causal discovery, by incorporating RL into the ordering-based paradigm. Specifically, we formulate the ordering search problem as a multi-step Markov decision process, implement the ordering generating process with an encoder-decoder architecture, and finally use RL to optimize the proposed model based on the reward mechanisms designed for~each ordering. A generated ordering would then be processed using variable selection to obtain the final causal graph. We analyze the consistency and computational complexity of the proposed method, and empirically show that a pretrained model can be exploited to accelerate training. Experimental results on both synthetic and real data sets shows that the proposed method achieves a much improved performance over existing RL-based method.