mlsanityData to 7 Oct 2026

Valley: Video Assistant with Large Language model Enhanced abilitY

arXiv 2306.07207 ↗First posted 12 Jun 2023cs.CV cs.AI cs.CLcited by 231

Not on the map. Why?

Cited by 105 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs), with remarkable conversational capability, have emerged as AI assistants that can handle both visual and textual modalities. However, their effectiveness in joint video and language understanding has not been extensively explored. In the paper, we introduce Valley, a multi-modal foundation model that is designed to enable enhanced video comprehension and instruction-following capabilities. To this end, we construct two datasets, namely Valley-702k and Valley-instruct-73k, to cover a diverse range of video-text alignment and video-based instruction tasks, such as multi-shot captions, long video descriptions, action recognition, causal inference, etc. Then, we adopt ViT-L/14 as the vision encoder and explore three different temporal modeling modules to learn multifaceted features for enhanced video understanding. In addition, we implement a two-phase training approach for Valley: the first phase focuses solely on training the projection module to facilitate the LLM's capacity to understand visual input, and the second phase jointly trains the projection module and the LLM to improve their instruction following ability. Extensive experiments demonstrate that Valley has the potential to serve as an effective video assistant, simplifying complex video-understanding scenarios. Our code and data are published anonymously at https://github.com/valley-vl/Valley.

Key facts. Cited by 231 papers in the map (data through 7 Oct 2026). Builds on 34 of its 56 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

34 of 56 references.

References · 34

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
22 more
  1. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  3. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  5. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,047
  6. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  7. 2023VideoChat: Chat-Centric Video Understandingcited by 851
  8. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  9. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  11. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  13. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  14. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  15. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  16. 2023PandaGPT: One Model To Instruction-Follow Them Allcited by 330
  17. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  18. 2022GRiT: A Generative Region-to-text Transformer for Object Understandingcited by 119
  19. 2023Tag2Text: Guiding Vision-Language Model via Image Taggingcited by 84
  20. 2023MemeCap: A Dataset for Captioning and Interpreting Memescited by 46
  21. 2020VIOLIN: A Large-Scale Dataset for Video-and-Language Inferencecited by 41
  22. 2020Learning to Discretely Compose Reasoning Module Networks for Video Captioningcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 25; 2024: 102; 2025: 81; 2026: 23.

231 papers.

Sort:

By area:

  1. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,385
  2. 2024Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysiscited by 1,375
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  4. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  5. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 539
  6. 2023DriveGPT4: Interpretable End-to-end Autonomous Driving via Large Language Modelcited by 455
  7. 2023TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understandingcited by 399
  8. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  9. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  10. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  11. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  12. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
38 more
  1. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 267
  2. 2023VTimeLLM: Empower LLM to Grasp Video Momentscited by 253
  3. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  4. 2024Video-XL: Extra-Long Vision Language Model for Hour-Scale Video Understandingcited by 210
  5. 2023Video Understanding with Large Language Models: A Surveycited by 203
  6. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  7. 2024InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Outputcited by 195
  8. 2024Streaming Long Video Understanding with Large Language Modelscited by 164
  9. 2024Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streamscited by 160
  10. 2025Adaptive Keyframe Sampling for Long Video Understandingcited by 153
  11. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  12. 2024Direct Preference Optimization of Video Large Multimodal Models from Language Model Rewardcited by 145
  13. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehensioncited by 144
  14. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  15. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  16. 2024ST-LLM: Large Language Models Are Effective Temporal Learnerscited by 126
  17. 2023Dolphins: Multimodal Language Model for Drivingcited by 124
  18. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  19. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  20. 2024Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuningcited by 117
  21. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  22. 2024HourVideo: 1-Hour Video-Language Understandingcited by 110
  23. 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
  24. 2023Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Modelscited by 107
  25. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Modelscited by 106
  26. 2024GroundingGPT:Language Enhanced Multi-modal Grounding Modelcited by 101
  27. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  28. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  29. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  30. 2024DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Modelscited by 93
  31. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 91
  32. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  33. 2024ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Modelscited by 89
  34. 2024VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Modelscited by 88
  35. 2024Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Modelscited by 87
  36. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  37. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 87
  38. 2024MileBench: Benchmarking MLLMs in Long Contextcited by 86

50 most cited of 231 All 231 (JSON)