mlsanityData to 7 Oct 2026

Octopus v3: Technical Report for On-device Sub-billion Multimodal AI Agent

arXiv 2404.11459 ↗First posted 17 Apr 2024cs.CL cs.CVcited by 7

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

A multimodal AI agent is characterized by its ability to process and learn from various types of data, including natural language, visual, and audio inputs, to inform its actions. Despite advancements in large language models that incorporate visual data, such as GPT-4V, effectively translating image-based data into actionable outcomes for AI agents continues to be challenging. In this paper, we introduce a multimodal model that incorporates the concept of functional token specifically designed for AI agent applications. To ensure compatibility with edge devices, our model is optimized to a compact size of less than 1B parameters. Like GPT-4, our model can process both English and Chinese. We demonstrate that this model is capable of operating efficiently on a wide range of edge devices, including as constrained as a Raspberry Pi.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 35 of its 54 references that are in the map. First posted on arXiv 17 Apr 2024.

Builds on

35 of 54 references.

References · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  10. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  11. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  12. 2021ViViT: A Video Vision Transformercited by 1,610
23 more
  1. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  4. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  7. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  8. 2020A Survey on Visual Transformercited by 653
  9. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  10. 2020A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospectscited by 396
  11. 2017Embodied Question Answeringcited by 335
  12. 2020Learning to Explore using Active Neural SLAMcited by 308
  13. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  14. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  15. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  16. 2023A Study of Generative Large Language Model for Medical Research and Healthcarecited by 160
  17. 2023Drive Like a Human: Rethinking Autonomous Driving with Large Language Modelscited by 158
  18. 2021Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAEcited by 106
  19. 2024Octopus v2: On-device language model for super agentcited by 45
  20. 2019Quantile QT-Opt for Risk-Aware Vision-Based Robotic Graspingcited by 27
  21. 2024Octopus: On-device language model for function calling of software APIscited by 26
  22. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  23. 2017Evaluating Visual Conversational Agents via Cooperative Human-AI Gamescited by 17

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 3.

7 papers.

  1. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  2. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  3. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  4. 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
  5. 2024Octopus v4: Graph of language modelscited by 11
  6. 2025Towards Pervasive Distributed Agentic Generative AI -- A State of The Artcited by 4
  7. 2025A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resourcescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Magma: A Foundation Model for Multimodal AI Agents
  2. 2026GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents
  3. 2026Qwen3.8-Omni: Towards Native Omni-Modal Agents
  4. 2025DeepEyesV2: Toward Agentic Multimodal Model
  5. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents
  6. 2024Agent AI: Surveying the Horizons of Multimodal Interaction
  7. 2025Agent-Omni: Test-Time Multimodal Reasoning via Model Coordination for Understanding Anything
  8. 2024Large Multimodal Agents: A Survey