mlsanityData to 14 Oct 2026

Exploring validation metrics for offline model-based optimisation with diffusion models

arXiv 2211.10747 ↗First posted 19 Nov 2022stat.ML cs.LGcited by 1

On the map: Diffusion models → Diffusion foundations and sampling

Abstract (arXiv)

In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle, which is expensive to compute since it involves executing a real world process. In offline MBO we wish to do so without assuming access to such an oracle during training or validation, with makes evaluation non-straightforward. While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples. Measuring the mean reward of generated candidates over this approximation is one such `validation metric', whereas we are interested in a more fundamental question which is finding which validation metrics correlate the most with the ground truth. This involves proposing validation metrics and quantifying them over many datasets for which the ground truth is known, for instance simulated environments. This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation, which is the ultimate goal behind leveraging generative models for MBO. While our evaluation framework is model agnostic we specifically evaluate denoising diffusion models due to their state-of-the-art performance, as well as derive interesting insights such as ranking the most effective validation metrics as well as discussing important hyperparameters.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Diffusion foundations and sampling, one of 14 directions in Diffusion models. Builds on 29 of its 38 references that are in the map. First posted on arXiv 19 Nov 2022.

Builds on

29 of 38 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Classifier-Free Diffusion Guidance26 Jul
  2. 2021Diffusion Models Beat GANs on Image Synthesis11 May
  3. 2020Denoising Diffusion Probabilistic Models19 Jun
  4. 2020Improved Techniques for Training Score-Based Generative Models16 Jun

Other references · 25

  1. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  2. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  3. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  4. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  5. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 434
  6. 2018Assessing Generative Models via Precision and Recallcited by 406
  7. 2015A note on the evaluation of generative modelscited by 292
  8. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  9. 2017GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibriumcited by 213
  10. 2021Pros and Cons of GAN Evaluation Measures: New Developmentscited by 123
  11. 2019Conditioning by adaptive sampling for robust designcited by 111
  12. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 97
13 more
  1. 2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 88
  2. 2019Model Inversion Networks for Model-Based Optimizationcited by 83
  3. 2022Local Latent Space Bayesian Optimization over Structured Inputscited by 79
  4. 2015How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary?cited by 78
  5. 2021Few-Shot Bayesian Optimization with Deep Kernel Surrogatescited by 71
  6. 2021Pre-trained Gaussian Processes for Bayesian Optimizationcited by 63
  7. 2020Autofocused oracles for model-based designcited by 50
  8. 2021RoMA: Robust Model Adaptation for Offline Model-based Optimizationcited by 39
  9. 2021Offline Model-Based Optimization via Normalized Maximum Likelihood Estimationcited by 38
  10. 2022Bidirectional Learning for Offline Infinite-width Model-based Optimizationcited by 35
  11. 2020Making Sense of Reinforcement Learning and Probabilistic Inferencecited by 30
  12. 2021JUMBO: Scalable Multi-task Bayesian Optimization using Offline Datacited by 8
  13. 2022Implicit Offline Reinforcement Learning via Supervised Learningcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024From Function to Distribution Modeling: A PAC-Generative Approach to Offline Optimizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Offline Model-Based Optimization: Comprehensive Review
  2. 2024Design Editing for Offline Model-based Optimization
  3. 2023Diffusion Models for Black-Box Optimization
  4. 2024Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimization
  5. 2026Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3
  6. 2026Learning Where to Steer: Noise-Space Geometry for Efficient Offline Multi-Objective Optimization with Generative Models
  7. 2022A Study on the Evaluation of Generative Models
  8. 2024Evaluating the design space of diffusion-based generative models