mlsanityData to 7 Oct 2026

CoCoG: Controllable Visual Stimuli Generation based on Human Concept Representations

arXiv 2404.16482 ↗First posted 25 Apr 2024q-bio.NC cs.CV cs.HCcited by 5

Not on the map. Why?

Abstract (arXiv)

A central question for cognitive science is to understand how humans process visual objects, i.e, to uncover human low-dimensional concept representation space from high-dimensional visual stimuli. Generating visual stimuli with controlling concepts is the key. However, there are currently no generative models in AI to solve this problem. Here, we present the Concept based Controllable Generation (CoCoG) framework. CoCoG consists of two components, a simple yet efficient AI agent for extracting interpretable concept and predicting human decision-making in visual similarity judgment tasks, and a conditional generation model for generating visual stimuli given the concepts. We quantify the performance of CoCoG from two aspects, the human behavior prediction accuracy and the controllable generation ability. The experiments with CoCoG indicate that 1) the reliable concept embeddings in CoCoG allows to predict human behavior with 64.07% accuracy in the THINGS-similarity dataset; 2) CoCoG can generate diverse objects through the control of concepts; 3) CoCoG can manipulate human similarity judgment behavior by intervening key concepts. CoCoG offers visual objects with controlling concepts to advance our understanding of causality in human cognition. The code of CoCoG is available at urlhttps://github.com/ncclab-sustech/CoCoG.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 36 of its 43 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

36 of 43 references.

References · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Classifier-Free Diffusion Guidancecited by 5,717
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  10. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  11. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  12. 2022Denoising Diffusion Restoration Modelscited by 951
24 more
  1. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 950
  2. 2023Adversarial Diffusion Distillationcited by 741
  3. 2021Solving Inverse Problems in Medical Imaging with Score-Based Generative Modelscited by 514
  4. 2022Improving Diffusion Models for Inverse Problems using Manifold Constraintscited by 501
  5. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  6. 2023Universal Guidance for Diffusion Modelscited by 426
  7. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  8. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 290
  9. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  10. 2022Unsupervised Medical Image Translation with Adversarial Diffusion Modelscited by 191
  11. 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
  12. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 144
  13. 2023Modern Bayesian Experimental Designcited by 121
  14. 2022Human alignment of neural network representationscited by 74
  15. 2023Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Modelscited by 65
  16. 2023Improving neural network representations using human similarity judgmentscited by 38
  17. 2023Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oraclescited by 34
  18. 2020Enriching ImageNet with Human Similarity Judgments and Psychological Embeddingscited by 23
  19. 2023FABRIC: Personalizing Diffusion Models with Iterative Feedbackcited by 19
  20. 2022Words are all you need? Language as an approximation for human similarity judgmentscited by 18
  21. 2019Revealing interpretable object representations from human behaviorcited by 17
  22. 2020Extracting low-dimensional psychological representations from convolutional neural networkscited by 16
  23. 2022VICE: Variational Interpretable Concept Embeddingscited by 15
  24. 2022Predicting Human Similarity Judgments Using Large Language Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 24
  2. 2024CoCoG-2: Controllable generation of visual stimuli for understanding human concept representationcited by 3
  3. 2025Synthesizing Images on Perceptual Boundaries of ANNs for Uncovering Human Perceptual Variability on Facial Expressionscited by 0
  4. 2025Dimensions of Vulnerability in Visual Working Memory: An AI-Driven Approach to Perceptual Comparisoncited by 0
  5. 2026MindPilot: Closed-loop Visual Stimulation Optimization for Brain Modulation with EEG-guided Diffusioncited by 0