mlsanityData to 7 Oct 2026

Words That Make Language Models Perceive

arXiv 2510.02425 ↗First posted 2 Oct 2025cs.CL cs.CV cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) trained purely on text ostensibly lack any direct perceptual experience, yet their internal representations are implicitly shaped by multimodal regularities encoded in language. We test the hypothesis that explicit sensory prompting can surface this latent structure, bringing a text-only LLM into closer representational alignment with specialist vision and audio encoders. When a sensory prompt tells the model to 'see' or 'hear', it cues the model to resolve its next-token predictions as if they were conditioned on latent visual or auditory evidence that is never actually supplied. Our findings reveal that lightweight prompt engineering can reliably activate modality-appropriate representations in purely text-trained LLMs.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 2 Oct 2025.

Builds on

20 of 32 references.

References · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  7. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  9. 2024Phi-4 Technical Reportcited by 675
  10. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  11. 2024The Platonic Representation Hypothesiscited by 362
  12. 2019Clotho: An Audio Captioning Datasetcited by 354
8 more
  1. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  2. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  3. 2022Masked Autoencoders that Listencited by 271
  4. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  5. 2021Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in Colorcited by 118
  6. 2023A Picture is Worth More Than 77 Text Tokens: Evaluating CLIP-Style Models on Dense Captionscited by 107
  7. 2022I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervisioncited by 26
  8. 2025LLMs can see and hear without any trainingcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.