mlsanityData to 7 Oct 2026

Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

arXiv 2403.05530 ↗First posted 8 Mar 2024cs.CL cs.AIcited by 3,175

Not on the map. Why?

Cited by 847 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.

Key facts. Cited by 3,175 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Mar 2024.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 870; 2025: 1660; 2026: 645.

3,175 papers.

Sort:

By area:

  1. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  2. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  5. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,371
  7. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  8. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phonecited by 1,061
  9. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,033
  10. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  11. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  12. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
38 more
  1. 2025Qwen2.5-Omni Technical Reportcited by 704
  2. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  3. 2024Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spacescited by 580
  4. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  5. 2024Qwen2-Audio Technical Reportcited by 543
  6. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  7. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  8. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
  9. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  10. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  11. 2025Humanity's Last Examcited by 467
  12. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  13. 2025Qwen3-Omni Technical Reportcited by 451
  14. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  15. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  16. 2024LVBench: An Extreme Long Video Understanding Benchmarkcited by 402
  17. 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 391
  18. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  19. 2025Kimi-VL Technical Reportcited by 381
  20. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  21. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  22. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  23. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  24. 2024What matters when building vision-language models?cited by 321
  25. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 321
  26. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  27. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  28. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 292
  29. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  30. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 273
  31. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 272
  32. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  33. 2025Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learningcited by 267
  34. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
  35. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  36. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  37. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  38. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242

50 most cited of 3,175 All 3,175 (JSON)