LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models
On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency
Abstract (arXiv)
Vision-Language Models (VLMs) typically assume a uniform spatial fidelity across the entire field of view of visual inputs, dedicating equal precision to even the uninformative regions. By contrast, human vision is neither uniform nor static; it is adaptive, selective, and resource-efficient. In light of this, we present the first systematic analysis of bio-inspired visual representation methods, providing insights for more efficient and adaptive VLMs. We propose LLMind (Looking Like the Mind), a novel training-free framework that mimics foveated encoding and cortical magnification in human vision to achieve adaptive, efficient representations for VLMs under tight pixel budgets. Our key idea is to explore a Bio-inspired Adaptive Sampling Strategy (BASS), enabling a Mobius-parameterized module that performs non-uniform sampling while preserving global scene structure. On top of BASS, we introduce closed-loop semantic feedback (CSF) via test-time adaptation to align perceptual saliency with textual information from the frozen VLM. We evaluate LLMind against uniform and other sampling baselines across diverse scene-level and region-guided visual question answering benchmarks. The results show dramatic gains, with average improvements of +20% on VQAv2, +38% on Seed-Bench, and +37% on A-OKVQA compared to uniform sampling under tight pixel budgets. More surprisingly, LLMind retains up to 82%, 92%, and 97% of the full-resolution performance using only 1%, 3%, and 5% of the pixels, respectively. Moreover, LLMind is lightweight, plug-and-play, and compatible with existing VLMs without requiring architectural changes.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 58 references that are in the map. First posted on arXiv 16 Mar 2026.
Builds on
36 of 58 references.
Closest ancestors · 1
Same direction, newest first
Other references · 35
- 2025Qwen3 Technical Reportcited by 8,195
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2022Visual Prompt Tuningcited by 1,674
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
- 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
- 2021Transformers in Vision: A Surveycited by 1,009
23 more
- 2022Token Merging: Your ViT But Fastercited by 995
- 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
- 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
- 2020Image Quality Assessment: Unifying Structure and Texture Similaritycited by 792
- 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
- 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
- 2020A Survey on Visual Transformercited by 653
- 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
- 2024FeatUp: A Model-Agnostic Framework for Features at Any Resolutioncited by 114
- 2024Mini-InternVL: A Flexible-Transfer Pocket Multimodal Model with 5% Parameters and 90% Performancecited by 94
- 2022A General Survey on Attention Mechanisms in Deep Learningcited by 64
- 2024ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Modelscited by 33
- 2022Peripheral Vision Transformercited by 27
- 2020Emergent Properties of Foveated Perceptual Systemscited by 22
- 2014Object Detection Through Exploration With A Foveated Visual Fieldcited by 14
- 2025Emulating Human-like Adaptive Vision for Efficient and Flexible Machine Visual Perceptioncited by 13
- 2017Central and peripheral vision for scene recognition: A neurocomputational modeling explorationcited by 8
- 2018Human peripheral blur is optimal for object recognitioncited by 8
- 2022Behind the Machine's Gaze: Neural Networks with Biologically-inspired Constraints Exhibit Human-like Visual Attentioncited by 7
- 2023Foveation in the Era of Deep Learningcited by 7
- 2025Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspectivecited by 7
- 2024ScribGen: Generating Scribble Art Through Metaheuristicscited by 3
- 2023Learning to search for and detect objects in foveal images using deep learningcited by 2
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rolloutcited by 6
- 2026Toward Native Multimodal Modeling: A Roadmapcited by 6
- 2026EAGOR: Embodied Reasoning in Omni-directioncited by 3
- 2026AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentationcited by 1
- 2026FAVE: Foveated Adaptive Visual Encoding for Efficient Fine-Grained Visual Understandingcited by 0
- 2026SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Cameracited by 0
- 2026SphMind: Towards Robust, Training-Free VLM-based Spatial Reasoning with a 360 Camera27 Sep 2026
- 2026FAVE: Foveated Adaptive Visual Encoding for Efficient Fine-Grained Visual Understanding3 Sep 2026
- 2026AdapToPASS: Ambiguity-aware Adaptive Spherical Transformer for Panoramic Semantic Segmentation29 Aug 2026
- 2026EAGOR: Embodied Reasoning in Omni-direction7 Jul 2026
- 2026Toward Native Multimodal Modeling: A Roadmap25 May 2026
- 2026Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout6 May 2026
Similar, not cited
Close in wording, no citation either way
- 2024Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inference
- 2025Adapting Vision-Language Models Without Labels: A Comprehensive Survey
- 2026Language-Instructed Vision Embeddings for Controllable and Generalizable Perception
- 2025Rethinking Visual Information Processing in Multimodal LLMs
- 2025Enhancing Self-Supervised Visual Representation Learning via Low-Rank Adapted LLMs
- 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models
- 2025Vision Language Models: A Survey of 26K Papers
- 2026PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference