mlsanityData to 14 Oct 2026

Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolution

arXiv 2409.12961 ↗First posted 19 Sep 2024cs.CVcited by 159

Not on the map. Why?

Cited by 85 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Visual data comes in various forms, ranging from small icons of just a few pixels to long videos spanning hours. Existing multi-modal LLMs usually standardize these diverse visual inputs to a fixed resolution for visual encoders and yield similar numbers of tokens for LLMs. This approach is non-optimal for multimodal understanding and inefficient for processing inputs with long and short visual contents. To solve the problem, we propose Oryx, a unified multimodal architecture for the spatial-temporal understanding of images, videos, and multi-view 3D scenes. Oryx offers an on-demand solution to seamlessly and efficiently process visual inputs with arbitrary spatial sizes and temporal lengths through two core innovations: 1) a pre-trained OryxViT model that can encode images at any resolution into LLM-friendly visual representations; 2) a dynamic compressor module that supports 1x to 16x compression on visual tokens by request. These design features enable Oryx to accommodate extremely long visual contexts, such as videos, with lower resolution and high compression while maintaining high recognition precision for tasks like document understanding with native resolution and no compression. Beyond the architectural improvements, enhanced data curation and specialized training on long-context retrieval and spatial-aware data help Oryx achieve strong capabilities in image, video, and 3D multimodal understanding simultaneously. Our work is open-sourced at https://github.com/Oryx-mllm/Oryx.

Key facts. Cited by 159 papers in the map (data through 14 Oct 2026). Builds on 62 of its 73 references that are in the map. First posted on arXiv 19 Sep 2024.

Builds on

62 of 73 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  9. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2019Towards VQA Models That Can Readcited by 1,718
  6. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,413
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  10. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,079
  11. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  13. 2016A Diagram Is Worth A Dozen Imagescited by 804
  14. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  15. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  16. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  17. 2024Yi: Open Foundation Models by 01.AIcited by 719
  18. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 665
  19. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  20. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  21. 2024Long Context Transfer from Language to Visioncited by 515
  22. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
  23. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 388
  24. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  25. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  26. 2024What matters when building vision-language models?cited by 326
  27. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  28. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 314
  29. 2024LongVILA: Scaling Long-Context Visual Language Models for Long Videoscited by 309
  30. 2023Track Anything: Segment Anything Meets Videoscited by 234
  31. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  32. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  33. 2020MovieNet: A Holistic Dataset for Movie Understandingcited by 206
  34. 2024LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Imagescited by 195
  35. 2024MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understandingcited by 182
  36. 2024VITA: Towards Open-Source Interactive Omni Multimodal LLMcited by 173
  37. 2023AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into Onecited by 169
  38. 2024Streaming Long Video Understanding with Large Language Modelscited by 167

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 20; 2025: 105; 2026: 34.

159 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  2. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  3. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  4. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  5. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  6. 2025Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligencecited by 217
  7. 2024NVILA: Efficient Frontier Visual Language Modelscited by 209
  8. 2025ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Drivingcited by 198
  9. 2025VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstructioncited by 166
  10. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 138
  11. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understandingcited by 138
  12. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
38 more
  1. 2024Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Modelscited by 116
  2. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  3. 2024Apollo: An Exploration of Video Understanding in Large Multimodal Modelscited by 89
  4. 2024CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understandingcited by 85
  5. 2025Ola: Pushing the Frontiers of Omni-Modal Language Modelcited by 83
  6. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 81
  7. 2025Flash-VStream: Efficient Real-Time Understanding for Long Video Streamscited by 81
  8. 2025StreamForest: Efficient Online Video Understanding with Persistent Event Memorycited by 78
  9. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 70
  10. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understandingcited by 70
  11. 2025V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoningcited by 68
  12. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 68
  13. 2025StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistantcited by 63
  14. 2025HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understandingcited by 60
  15. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  16. 2025EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agentscited by 57
  17. 2025SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understandingcited by 57
  18. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 49
  19. 2025Video-XL-Pro: Reconstructive Token Compression for Extremely Long Video Understandingcited by 49
  20. 2025Open-o3-Video: Grounded Video Reasoning with Explicit Spatio-Temporal Evidencecited by 49
  21. 2025MDP3: A Training-free Approach for List-wise Frame Selection in Video-LLMscited by 43
  22. 2025ViSpeak: Visual Instruction Feedback in Streaming Videoscited by 42
  23. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 39
  24. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 37
  25. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 36
  26. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  27. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  28. 20253D Aware Region Prompted Vision Language Modelcited by 33
  29. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  30. 2025UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generationcited by 32
  31. 2025SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understandingcited by 30
  32. 2025AToken: A Unified Tokenizer for Visioncited by 30
  33. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 29
  34. 2025VideoITG: Multimodal Video Understanding with Instructed Temporal Groundingcited by 29
  35. 2024Robin3D: Improving 3D Large Language Model via Robust Instruction Tuningcited by 26
  36. 2025LLaVA-Scissor: Token Compression with Semantic Connected Components for Video LLMscited by 26
  37. 2025Streaming Video Instruction Tuningcited by 26
  38. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25

50 most cited of 159 All 159 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PixelUMM: Encoder-Free Unified Image and Video Understanding and Generation
  2. 2025ToFu: Visual Tokens Reduction via Fusion for Multi-modal, Multi-patch, Multi-image Task
  3. 2026Qwen-3D: A Generalist 3D Vision-Language Model for Spatial Understanding
  4. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation
  5. 2023OtterHD: A High-Resolution Multi-modality Model
  6. 2026Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors
  7. 2026Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification
  8. 2026ApET: Approximation-Error Guided Token Compression for Efficient VLMs