mlsanityData to 7 Oct 2026

QR-CLIP: Introducing Explicit Open-World Knowledge for Location and Time Reasoning

arXiv 2302.00952 ↗First posted 2 Feb 2023cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Daily images may convey abstract meanings that require us to memorize and infer profound information from them. To encourage such human-like reasoning, in this work, we teach machines to predict where and when it was taken rather than performing basic tasks like traditional segmentation or classification. Inspired by Horn's QR theory, we designed a novel QR-CLIP model consisting of two components: 1) the Quantity module first retrospects more open-world knowledge as the candidate language inputs; 2) the Relevance module carefully estimates vision and language cues and infers the location and time. Experiments show our QR-CLIP's effectiveness, and it outperforms the previous SOTA on each task by an average of about 10% and 130% relative lift in terms of location and time reasoning. This study lays a technical foundation for location and time reasoning and suggests that effectively introducing open-world knowledge is one of the panaceas for the tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 50 of its 76 references that are in the map. First posted on arXiv 2 Feb 2023.

Builds on

50 of 76 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2022Training language models to follow instructions with human feedbackcited by 15,712
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  13. 2022A Generalist Agentcited by 802
  14. 2022Point-E: A System for Generating 3D Point Clouds from Complex Promptscited by 694
  15. 2019Green AIcited by 620
  16. 2020Jukebox: A Generative Model for Musiccited by 535
  17. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  18. 2022Scaling Vision Transformers to Gigapixel Images via Hierarchical Self-Supervised Learningcited by 357
  19. 2021WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learningcited by 285
  20. 2022A Contrastive Framework for Neural Text Generationcited by 224
  21. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  22. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 89
  23. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  24. 2020Multi-document Summarization via Deep Learning Techniques: A Surveycited by 58
  25. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Modelscited by 52
  26. 2022Multi-View Document Representation Learning for Open-Domain Dense Retrievalcited by 41
  27. 2021N24News: A New Dataset for Multimodal News Classificationcited by 40
  28. 2020Temporal Common Sense Acquisition with Minimal Supervisioncited by 38
  29. 2020ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoningcited by 38
  30. 2022There is a Time and Place for Reasoning Beyond the Imagecited by 22
  31. 2022SimAN: Exploring Self-Supervised Representation Learning of Scene Text via Similarity-Aware Normalizationcited by 21
  32. 2022Masked Vision-Language Transformer in Fashioncited by 14
  33. 2022A study on the distribution of social biases in self-supervised learning visual modelscited by 13
  34. 2021Self-Supervised Learning by Estimating Twin Class Distributionscited by 12
  35. 2021GeoWINE: Geolocation based Wiki, Image,News and Event Retrievalcited by 4
  36. 2022COFAR: Commonsense and Factual Reasoning in Image Searchcited by 4
  37. 2019Learning Shared Semantic Space with Correlation Alignment for Cross-modal Event Retrievalcited by 3
  38. 2021Self-Supervised Learning of Perceptually Optimized Block Motion Estimates for Video Compressioncited by 2

Built on by

Not cited yet.