# Data-Driven Offline Decision-Making via Invariant Representation Learning

arXiv: https://arxiv.org/abs/2211.11349 · HTML: https://mlsanity.com/paper/2211.11349/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 30 of 35)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1910.00177/) (2019)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [COMBO: Conservative Offline Model-Based Policy Optimization](https://mlsanity.com/paper/2102.08363/) (2021)
- [Is Pessimism Provably Efficient for Offline RL?](https://mlsanity.com/paper/2012.15085/) (2020)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Scalable Bayesian Optimization Using Deep Neural Networks](https://mlsanity.com/paper/1502.05700/) (2015)
- [Safe Policy Improvement with Baseline Bootstrapping](https://mlsanity.com/paper/1712.06924/) (2017)
- [Conditioning by adaptive sampling for robust design](https://mlsanity.com/paper/1901.10060/) (2019)
- [Adversarially Trained Actor Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2202.02446/) (2022)
- [Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimization](https://mlsanity.com/paper/2202.08450/) (2022)
- [Conservative Objective Models for Effective Offline Model-Based Optimization](https://mlsanity.com/paper/2107.06882/) (2021)
- [Model Inversion Networks for Model-Based Optimization](https://mlsanity.com/paper/1912.13464/) (2019)
- [Chip Placement with Deep Reinforcement Learning](https://mlsanity.com/paper/2004.10746/) (2020)
- [The reparameterization trick for acquisition functions](https://mlsanity.com/paper/1712.00424/) (2017)
- [Distributionally Robust Bayesian Optimization](https://mlsanity.com/paper/2002.09038/) (2020)
- [Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RL](https://mlsanity.com/paper/2012.08005/) (2020)
- [Autofocused oracles for model-based design](https://mlsanity.com/paper/2006.08052/) (2020)
- [Learning Weighted Representations for Generalization Across Designs](https://mlsanity.com/paper/1802.08598/) (2018)
- [RoMA: Robust Model Adaptation for Offline Model-based Optimization](https://mlsanity.com/paper/2110.14188/) (2021)
- [Offline Model-Based Optimization via Normalized Maximum Likelihood Estimation](https://mlsanity.com/paper/2102.07970/) (2021)
- [Population-Based Black-Box Optimization for Biological Sequence Design](https://mlsanity.com/paper/2006.03227/) (2020)
- [Data-Driven Offline Optimization For Architecting Hardware Accelerators](https://mlsanity.com/paper/2110.11346/) (2021)

## Built on by (most cited first, 30 of 31)

- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [ParetoFlow: Guided Flows in Multi-Objective Optimization](https://mlsanity.com/paper/2412.03718/) (2024)
- [Offline Model-Based Optimization: Comprehensive Review](https://mlsanity.com/paper/2503.17286/) (2025)
- [Offline Multi-Objective Optimization](https://mlsanity.com/paper/2406.03722/) (2024)
- [Offline Model-Based Optimization by Learning to Rank](https://mlsanity.com/paper/2410.11502/) (2024)
- [Preference-Guided Diffusion for Multi-Objective Offline Optimization](https://mlsanity.com/paper/2503.17299/) (2025)
- [Functional Graphical Models: Structure Enables Offline Data-Driven Optimization](https://mlsanity.com/paper/2401.05442/) (2024)
- [Design Editing for Offline Model-based Optimization](https://mlsanity.com/paper/2405.13964/) (2024)
- ["Noisier" Noise Contrastive Eestimation is (Almost) Maximum Likelihood](https://mlsanity.com/paper/2405.16730/) (2024)
- [Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learning](https://mlsanity.com/paper/2305.15260/) (2023)
- [Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddings](https://mlsanity.com/paper/2506.07109/) (2025)
- [Latent Conservative Objective Models for Data-Driven Crystal Structure Prediction](https://mlsanity.com/paper/2310.10056/) (2023)
- [ROOT: Rethinking Offline Optimization as Distributional Translation via Probabilistic Bridge](https://mlsanity.com/paper/2509.16300/) (2025)
- [When is Offline Policy Selection Sample Efficient for Reinforcement Learning?](https://mlsanity.com/paper/2312.02355/) (2023)
- [Cliqueformer: Model-Based Optimization with Structured Transformers](https://mlsanity.com/paper/2410.13106/) (2024)
- [Out-of-Distribution Adaptation in Offline RL: Counterfactual Reasoning via Causal Normalizing Flows](https://mlsanity.com/paper/2405.03892/) (2024)
- [Sharpness-Aware Black-Box Optimization](https://mlsanity.com/paper/2410.12457/) (2024)
- [SPREAD: Sampling-based Pareto front Refinement via Efficient Adaptive Diffusion](https://mlsanity.com/paper/2509.21058/) (2025)
- [On the Learnability of Offline Model-Based Optimization: A Ranking Perspective](https://mlsanity.com/paper/2603.04000/) (2026)
- [ROMO: Retrieval-enhanced Offline Model-based Optimization](https://mlsanity.com/paper/2310.07560/) (2023)
- [From Function to Distribution Modeling: A PAC-Generative Approach to Offline Optimization](https://mlsanity.com/paper/2401.02019/) (2024)
- [Learning Design-Score Manifold to Guide Diffusion Models for Offline Optimization](https://mlsanity.com/paper/2506.05680/) (2025)
- [Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RL](https://mlsanity.com/paper/2511.03695/) (2025)
- [Conformal Candidate Certification for Offline Model-Based Optimization](https://mlsanity.com/paper/2606.15217/) (2026)
- [Conservative quantum offline model-based optimization](https://mlsanity.com/paper/2506.19714/) (2025)
- [Sample Efficient Generative Molecular Optimization with Joint Self-Improvement](https://mlsanity.com/paper/2602.10984/) (2026)
- [The Offline-Frontier Shift: Diagnosing Distributional Limits in Generative Multi-Objective Optimization](https://mlsanity.com/paper/2602.11126/) (2026)
- [ParetoPilot: Zero-Surrogate Offline Multi-Objective Optimization via Infer-Perturb-Guide Diffusion](https://mlsanity.com/paper/2606.04468/) (2026)
- [Generative Refinement for Low-Budget Black-Box Optimization](https://mlsanity.com/paper/2607.00691/) (2026)
- [Rethinking Learnability in Offline Data-driven Optimization](https://mlsanity.com/paper/2609.01493/) (2026)

## Abstract (arXiv)

The goal in offline data-driven decision-making is synthesize decisions that optimize a black-box utility function, using a previously-collected static dataset, with no active interaction. These problems appear in many forms: offline reinforcement learning (RL), where we must produce actions that optimize the long-term reward, bandits from logged data, where the goal is to determine the correct arm, and offline model-based optimization (MBO) problems, where we must find the optimal design provided access to only a static dataset. A key challenge in all these settings is distributional shift: when we optimize with respect to the input into a model trained from offline data, it is easy to produce an out-of-distribution (OOD) input that appears erroneously good. In contrast to prior approaches that utilize pessimism or conservatism to tackle this problem, in this paper, we formulate offline data-driven decision-making as domain adaptation, where the goal is to make accurate predictions for the value of optimized decisions ("target domain"), when training only on the dataset ("source domain"). This perspective leads to invariant objective models (IOM), our approach for addressing distributional shift by enforcing invariance between the learned representations of the training dataset and optimized decisions. In IOM, if the optimized decisions are too different from the training dataset, the representation will be forced to lose much of the information that distinguishes good designs from bad ones, making all choices seem mediocre. Critically, when the optimizer is aware of this representational tradeoff, it should choose not to stray too far from the training distribution, leading to a natural trade-off between distributional shift and learning performance.