mlsanityData to 14 Oct 2026

Cliqueformer: Model-Based Optimization with Structured Transformers

arXiv 2410.13106 ↗First posted 17 Oct 2024cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Large neural networks excel at prediction tasks, but their application to design problems, such as protein engineering or materials discovery, requires solving offline model-based optimization (MBO) problems. While predictive models may not directly translate to effective design, recent MBO algorithms incorporate reinforcement learning and generative modeling approaches. Meanwhile, theoretical work suggests that exploiting the target function's structure can enhance MBO performance. We present Cliqueformer, a transformer-based architecture that learns the black-box function's structure through functional graphical models (FGM), addressing distribution shift without relying on explicit conservative approaches. Across various domains, including chemical and genetic design tasks, Cliqueformer demonstrates superior performance compared to existing methods.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 28 of its 45 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

28 of 45 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  7. 2017Decoupled Weight Decay Regularizationcited by 1,831
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2016Deep Variational Information Bottleneckcited by 930
  10. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 765
  11. 2019Diagnosing and Enhancing VAE Modelscited by 193
  12. 2015High Dimensional Bayesian Optimisation and Bandits via Additive Modelscited by 139
16 more
  1. 2019Conditioning by adaptive sampling for robust designcited by 111
  2. 2024Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Reviewcited by 98
  3. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 97
  4. 2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 88
  5. 2019Model Inversion Networks for Model-Based Optimizationcited by 83
  6. 2023Diffusion Models for Black-Box Optimizationcited by 76
  7. 2018High-Dimensional Bayesian Optimization via Additive Models with Overlapping Groupscited by 59
  8. 2020Autofocused oracles for model-based designcited by 50
  9. 2022Generative Pretraining for Black-Box Optimizationcited by 36
  10. 2023Are Random Decompositions all we need in High Dimensional Bayesian Optimisation?cited by 36
  11. 2022Data-Driven Offline Decision-Making via Invariant Representation Learningcited by 31
  12. 2021Data-Driven Offline Optimization For Architecting Hardware Acceleratorscited by 29
  13. 2024Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Modelscited by 25
  14. 2024Generative Hierarchical Materials Searchcited by 13
  15. 2024Functional Graphical Models: Structure Enables Offline Data-Driven Optimizationcited by 11
  16. 2018Feedback GAN (FBGAN) for DNA: a Novel Feedback-Loop Architecture for Optimizing Protein Functionscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Offline Model-Based Optimization: Comprehensive Reviewcited by 15
  2. 2025Towards Universal Offline Black-Box Optimization via Learning Language Model Embeddingscited by 7
  3. 2026Offline Materials Optimization with CliqueFlowmercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Design Editing for Offline Model-based Optimization
  2. 2026A Closer Look at Agentic BBO: Benchmarking LLM Agents for Black-Box Optimization
  3. 2024Model-based reinforcement learning for protein backbone design
  4. 2026Purely Agent-Driven Black-Box Optimization for Biological Design
  5. 2025Learning Design-Score Manifold to Guide Diffusion Models for Offline Optimization
  6. 2026Constrained Flow Optimization via Sequential Fine Tuning for Molecular Design
  7. 2026Structure-aware Reinforcement Learning for Protein Directed Evolution
  8. 2024Offline Model-Based Optimization via Policy-Guided Gradient Search