mlsanityData to 7 Oct 2026

Olympus: A Universal Task Router for Computer Vision Tasks

arXiv 2412.09612 ↗First posted 12 Dec 2024cs.CV cs.AI cs.CLcited by 4

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

We introduce Olympus, a new approach that transforms Multimodal Large Language Models (MLLMs) into a unified framework capable of handling a wide array of computer vision tasks. Utilizing a controller MLLM, Olympus delegates over 20 specialized tasks across images, videos, and 3D objects to dedicated modules. This instruction-based routing enables complex workflows through chained actions without the need for training heavy generative models. Olympus easily integrates with existing MLLMs, expanding their capabilities with comparable performance. Experimental results demonstrate that Olympus achieves an average routing accuracy of 94.75% across 20 tasks and precision of 91.82% in chained action scenarios, showcasing its effectiveness as a universal task router that can solve a diverse range of computer vision tasks. Project page: http://yuanze-lin.me/Olympus_page/

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 87 of its 100 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

87 of 100 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  2. 2024MobileVLM V2: Faster and Stronger Baseline for Vision Language Model6 Feb
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 83

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  11. 2024GPT-4o System Cardcited by 4,467
  12. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
38 more
  1. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  2. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  7. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
  13. 2022Video Diffusion Modelscited by 2,280
  14. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
  15. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  16. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  17. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  18. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  19. 2019Towards VQA Models That Can Readcited by 1,696
  20. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  21. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  22. 2024Depth Anything V2cited by 1,385
  23. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  24. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  25. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  26. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  27. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  28. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  29. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  30. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  31. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  32. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  33. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  34. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  35. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  36. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  37. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  38. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545

50 most cited of 83 All 83 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  2. 2025IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generationcited by 6
  3. 2025SoilNet: A Multimodal Multitask Model for Hierarchical Classification of Soil Horizonscited by 2
  4. 2026Continual Model Routing in Evolving Model Hubscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025ECVL-ROUTER: Scenario-Aware Routing for Vision-Language Models
  2. 2025VL-RouterBench: A Benchmark for Vision-Language Model Routing
  3. 2023RoboLLM: Robotic Vision Tasks Grounded on Multimodal Large Language Models
  4. 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing
  5. 2025NanoVLA: Routing Decoupled Vision-Language Understanding for Nano-sized Generalist Robotic Policies
  6. 2024SOLO: A Single Transformer for Scalable Vision-Language Modeling
  7. 2026VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization
  8. 2025Long-Tailed Distribution-Aware Router For Mixture-of-Experts in Large Vision-Language Model