mlsanityData to 7 Oct 2026

Large language models predict human sensory judgments across six modalities

arXiv 2302.01308 ↗First posted 2 Feb 2023cs.CL cs.LG stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Determining the extent to which the perceptual world can be recovered from language is a longstanding problem in philosophy and cognitive science. We show that state-of-the-art large language models can unlock new insights into this problem by providing a lower bound on the amount of perceptual information that can be extracted from language. Specifically, we elicit pairwise similarity judgments from GPT models across six psychophysical datasets. We show that the judgments are significantly correlated with human data across all domains, recovering well-known representations like the color wheel and pitch spiral. Surprisingly, we find that a model (GPT-4) co-trained on vision and language does not necessarily lead to improvements specific to the visual modality. To study the influence of specific languages on perception, we also apply the models to a multilingual color-naming task. We find that GPT-4 replicates cross-linguistic variation in English and Russian illuminating the interaction of language and perception.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 8 of its 62 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

8 of 62 references.

References · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2022Using cognitive psychology to understand GPT-3cited by 389
  4. 2022Emergent Analogical Reasoning in Large Language Modelscited by 239
  5. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  6. 2022Visual Commonsense in Pretrained Unimodal and Multimodal Modelscited by 39
  7. 2020Gibbs Sampling with Peoplecited by 28
  8. 2018Generative timbre spaces: regularizing variational auto-encoders with perceptual metricscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 19; 2026: 10.

47 papers.

Sort:

By area:

  1. 2024Mind Your Step (by Step): Chain-of-Thought can Reduce Performance on Tasks where Thinking Makes Humans Worsecited by 107
  2. 2024Attention Heads of Large Language Models: A Surveycited by 65
  3. 2024One fish, two fish, but not the whole sea: Alignment reduces language models' conceptual diversitycited by 57
  4. 2024Large Language Models and Cognitive Science: A Comprehensive Review of Similarities, Differences, and Challengescited by 48
  5. 2024Human-like object concept representations emerge naturally in multimodal large language modelscited by 24
  6. 2024Exploring the Frontiers of LLMs in Psychological Applications: A Comprehensive Reviewcited by 17
  7. 2024What Do Language Models Hear? Probing for Auditory Representations in Language Modelscited by 15
  8. 2025What is a Number, That a Large Language Model May Know It?cited by 14
  9. 2025The threat of analytic flexibility in using large language models to simulate human datacited by 13
  10. 2024Language Models Trained to do Arithmetic Predict Human Risky and Intertemporal Choicecited by 10
  11. 2025Levels of Analysis for Large Language Modelscited by 9
  12. 2024How Well Do Deep Learning Models Capture Human Concepts? The Case of the Typicality Effectcited by 7
35 more
  1. 2024A Comparison of Large Language Model and Human Performance on Random Number Generation Taskscited by 7
  2. 2025Using LLMs to Advance the Cognitive Science of Collectivescited by 6
  3. 2025Large Language Models as Psychological Simulators: A Methodological Guidecited by 6
  4. 2024Constructive Approach to Bidirectional Influence between Qualia Structure and Language Emergencecited by 5
  5. 2025The Zero Body Problem: Probing LLM Use of Sensory Languagecited by 5
  6. 2025Steering Risk Preferences in Large Language Models by Aligning Behavioral and Neural Representationscited by 5
  7. 2025CHART-6: Human-Centered Evaluation of Data Visualization Understanding in Vision-Language Modelscited by 5
  8. 2026Post-training makes large language models less human-likecited by 4
  9. 2024Characterizing Similarities and Divergences in Conversational Tones in Humans and LLMs by Sampling with Peoplecited by 3
  10. 2025On Entity Identification in Language Modelscited by 3
  11. 2025Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AIcited by 3
  12. 2026Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Modelscited by 3
  13. 2026Benchmark for Assessing Olfactory Perception of Large Language Modelscited by 3
  14. 2024Reflectance Estimation for Proximity Sensing by Vision-Language Models: Utilizing Distributional Semantics for Low-Level Cognition in Roboticscited by 2
  15. 2024A Nurse is Blue and Elephant is Rugby: Cross Domain Alignment in Large Language Models Reveal Human-like Patternscited by 2
  16. 2024TouchAI: Exploring human-AI perceptual alignment in touch through language model representationscited by 2
  17. 2025AI-enhanced semantic feature norms for 786 conceptscited by 2
  18. 2025How important is language for human-like intelligence?cited by 2
  19. 2025Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Modelscited by 2
  20. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understandingcited by 2
  21. 2024Advancements and limitations of LLMs in replicating human color-word associationscited by 1
  22. 2024A Flexible Method for Behaviorally Measuring Alignment Between Human and Artificial Intelligence Using Representational Similarity Analysiscited by 1
  23. 2025Correspondence of high-dimensional emotion structures elicited by video clips between humans and Multimodal LLMscited by 1
  24. 2025Evolution and compression in LLMs: On the emergence of human-aligned categorizationcited by 1
  25. 2026Geometry of Human Perceptual Domains Emerges Transiently in LLM Representationscited by 1
  26. 2023Are we describing the same sound? An analysis of word embedding spaces of expressive piano performancecited by 0
  27. 2025Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Modelcited by 0
  28. 2025Engineering Sentiencecited by 0
  29. 2025Extracting Conceptual Spaces from LLMs Using Prototype Embeddingscited by 0
  30. 2025Behavioral Geometric Supervision Aligns Video Foundation Models with Human Social Perceptioncited by 0
  31. 2026LORE: Jointly Learning the Intrinsic Dimensionality and Relative Similarity Structure From Ordinal Datacited by 0
  32. 2026The Image Reconstruction Game: Drawing Common Ground Through Iterative Multimodal Dialoguecited by 0
  33. 2026Do LLM Embedding Spaces Recover Expert Structure?cited by 0
  34. 2026Billions of Sketches Reveal Hidden Cultural Variation in Human Conceptscited by 0
  35. 2026Probing Perceptual Priors of MLLMs via Gibbs Sampling with Interpretable Generative Controlscited by 0