mlsanityData to 14 Oct 2026

GRIT: General Robust Image Task Benchmark

arXiv 2204.13653 ↗First posted 28 Apr 2022cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

Computer vision models excel at making predictions when the test distribution closely resembles the training distribution. Such models have yet to match the ability of biological vision to learn from multiple sources and generalize to new data sources and tasks. To facilitate the development and evaluation of more general vision systems, we introduce the General Robust Image Task (GRIT) benchmark. GRIT evaluates the performance, robustness, and calibration of a vision system across a variety of image prediction tasks, concepts, and data sources. The seven tasks in GRIT are selected to cover a range of visual skills: object categorization, object localization, referring expression grounding, visual question answering, segmentation, human keypoint detection, and surface normal estimation. GRIT is carefully designed to enable the evaluation of robustness under image perturbations, image source distribution shift, and concept distribution shift. By providing a unified platform for thorough assessment of skills and concepts learned by a vision model, we hope GRIT catalyzes the development of performant and robust general purpose vision systems.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Builds on 24 of its 38 references that are in the map. First posted on arXiv 28 Apr 2022.

Builds on

24 of 38 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2017Mask R-CNNcited by 7,277
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2015VQA: Visual Question Answeringcited by 2,506
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
12 more
  1. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  2. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  3. 2016Universal adversarial perturbationscited by 784
  4. 2019BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networkscited by 490
  5. 2014Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecturecited by 483
  6. 2021Boundary IoU: Improving Object-Centric Image Segmentation Evaluationcited by 188
  7. 2021Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimationcited by 116
  8. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  9. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  10. 2021Towards General Purpose Vision Systemscited by 46
  11. 2022Webly Supervised Concept Expansion for General Purpose Vision Modelscited by 45
  12. 2019FrameNet: Learning Local Canonical Frames of 3D Surfaces from a Single RGB Imagecited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 8; 2024: 7; 2025: 6; 2026: 2.

26 papers.

Sort:

By area:

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  2. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  3. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 302
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  6. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspectivecited by 75
  7. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 52
  8. 2024MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Modelscited by 34
  9. 2024MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learningcited by 30
  10. 2025ColorBench: Can VLMs See and Understand the Colorful World? A Comprehensive Benchmark for Color Perception, Reasoning, and Robustnesscited by 28
  11. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 25
  12. 2024μ-Bench: A Vision-Language Benchmark for Microscopy Understandingcited by 15
14 more
  1. 2023A Large-scale Study of Spatiotemporal Representation Learning with a New Benchmark on Action Recognitioncited by 14
  2. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Modelscited by 14
  3. 2024Visual Robustness Benchmark for Visual Question Answering (VQA)cited by 12
  4. 2022Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representationcited by 7
  5. 2023RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Modelscited by 7
  6. 2025MMPerspective: Do MLLMs Understand Perspective? A Comprehensive Benchmark for Perspective Perception, Reasoning, and Robustnesscited by 5
  7. 2025Re-Aligning Language to Visual Objects with an Agentic Workflowcited by 4
  8. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  9. 2026DroneFINE: Domain-Aware Parameter-Efficient Fine-Tuning of Vision-Language Detectors for Drone Imagescited by 3
  10. 2023Exposing and Addressing Cross-Task Inconsistency in Unified Vision-Language Modelscited by 2
  11. 2025Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Groundingcited by 1
  12. 2026Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluationcited by 1
  13. 2024Discerning the Chaos: Detecting Adversarial Perturbations while Disentangling Intentional from Unintentional Noisescited by 0
  14. 2025Vision Generalist Model: A Surveycited by 0