mlsanityData to 14 Oct 2026

Memory-Conditioned Tool Calling for Camera-First Visual Agents

arXiv 2607.09822 ↗First posted 10 Jul 2026cs.CV cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Recognition tells an agent what is in an image; personal memory affects what is worth looking up next. In a camera-first setting the user can send only an image, so the agent must form the lookups. We study whether personal visual memory improves agent-side tool choice and tool arguments, and thereby more user-aligned multi-tool lookups. The design uses a three-layer personal visual memory (profile, short-term focus, observations) that is loaded on each turn to condition an LLM tool-calling loop under camera-first intake, and includes conflict-aware write-back intended to refresh the user model for later captures. On 800 images paired with synthetic memory blocks constructed for controlled ablation, removing the full three-layer memory block reduces tool-query relevance by 0.47 points absolute (4.21 -> 3.74 on a 5-point scale; 11.2% relative) and end-to-end utility by 0.082 absolute (0.842 -> 0.760; 9.7% relative). These results measure memory conditioning of tool policy under image-only intake with fixed synthetic blocks, not multi-session write-back from live user histories.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 15 of its 20 references that are in the map. First posted on arXiv 10 Jul 2026.

Builds on

15 of 20 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory28 Apr
  2. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 13

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2017Billion-scale similarity search with GPUscited by 2,497
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  7. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  8. 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactionscited by 25
  9. 2026PersonaVLM: Long-Term Personalized Multimodal LLMscited by 11
  10. 2024RAVEN: Multitask Retrieval Augmented Vision-Language Learningcited by 10
  11. 2026MTA-Agent: An Open Recipe for Multimodal Deep Search Agentscited by 7
  12. 2026ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answeringcited by 5
1 more
  1. 2026VideoSearcher: Empowering Video Deep Research with Multi-Tool Agentic Reasoning via Reinforcement Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Memory-Induced Tool-Drift in LLM Agents
  2. 2026Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains
  3. 2026Latent Preference Modeling for Multi-Session Personalized Tool Calling
  4. 2025ToolMem: Enhancing Multimodal Agents with Learnable Tool Capability Memory
  5. 2026VC-Tooler: Learning Compositional and Adaptive Visual Tool Use
  6. 2026MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents
  7. 2026Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents
  8. 2026Looking Is Not Picking: An Attention-Segment Account of Tool-Selection Failures in LLM Agents