Words That Make Language Models Perceive
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) trained purely on text ostensibly lack any direct perceptual experience, yet their internal representations are implicitly shaped by multimodal regularities encoded in language. We test the hypothesis that explicit sensory prompting can surface this latent structure, bringing a text-only LLM into closer representational alignment with specialist vision and audio encoders. When a sensory prompt tells the model to 'see' or 'hear', it cues the model to resolve its next-token predictions as if they were conditioned on latent visual or auditory evidence that is never actually supplied. Our findings reveal that lightweight prompt engineering can reliably activate modality-appropriate representations in purely text-trained LLMs.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 2 Oct 2025.
Builds on
20 of 32 references.
References · 20
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2025Qwen3 Technical Reportcited by 8,195
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
- 2024Phi-4 Technical Reportcited by 675
- 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
- 2024The Platonic Representation Hypothesiscited by 362
- 2019Clotho: An Audio Captioning Datasetcited by 354
8 more
- 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
- 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
- 2022Masked Autoencoders that Listencited by 271
- 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
- 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
- 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
- 2022I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervisioncited by 26
- 2025LLMs can see and hear without any trainingcited by 8
Built on by
Citing papers per year: 2025: 1; 2026: 6.
7 papers.
- 2025Large Language Models Report Subjective Experience Under Self-Referential Processingcited by 16
- 2026Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scalecited by 10
- 2026The Truth Lies Somewhere in the Middle (of the Generated Tokens)cited by 2
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipescited by 1
- 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
- 2026Learning to Draw ASCII Improves Spatial Reasoning in Language Modelscited by 0
- 2026Displacement Geometry Captures Platonic Shared Reality Across Models and Modalitiescited by 0
- 2026Displacement Geometry Captures Platonic Shared Reality Across Models and Modalities21 Sep 2026
- 2026Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes5 Aug 2026
- 2026The Truth Lies Somewhere in the Middle (of the Generated Tokens)11 May 2026
- 2026Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale20 Apr 2026
- 2026Learning to Draw ASCII Improves Spatial Reasoning in Language Models16 Apr 2026
- 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergy8 Jan 2026
- 2025Large Language Models Report Subjective Experience Under Self-Referential Processing27 Oct 2025