# T-Rex: Counting by Visual Prompting

arXiv: https://arxiv.org/abs/2311.13596 · HTML: https://mlsanity.com/paper/2311.13596/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 52 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 30 of 44)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [Semantic-SAM: Segment and Recognize Anything at Any Granularity](https://mlsanity.com/paper/2307.04767/) (2023)
- [Drone-based Object Counting by Spatially Regularized Regional Proposal Network](https://mlsanity.com/paper/1707.05972/) (2017)
- [Learning To Count Everything](https://mlsanity.com/paper/2104.08391/) (2021)
- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)

## Built on by (most cited first, 20 of 20)

- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy](https://mlsanity.com/paper/2403.14610/) (2024)
- [DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding](https://mlsanity.com/paper/2411.14347/) (2024)
- [Detect Anything via Next Point Prediction](https://mlsanity.com/paper/2510.12798/) (2025)
- [YOLOE: Real-Time Seeing Anything](https://mlsanity.com/paper/2503.07465/) (2025)
- [Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring](https://mlsanity.com/paper/2403.09333/) (2024)
- [Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning](https://mlsanity.com/paper/2506.04034/) (2025)
- [CrowdVLM-R1: Expanding R1 Ability to Vision Language Model for Crowd Counting using Fuzzy Group Relative Policy Reward](https://mlsanity.com/paper/2504.03724/) (2025)
- [OpenRSD: Towards Open-prompts for Object Detection in Remote Sensing Images](https://mlsanity.com/paper/2503.06146/) (2025)
- [DINO-R1: Incentivizing Reasoning Capability in Vision Foundation Models](https://mlsanity.com/paper/2505.24025/) (2025)
- [LOCR: Location-Guided Transformer for Optical Character Recognition](https://mlsanity.com/paper/2403.02127/) (2024)
- [PET-DINO: Unifying Visual Cues into Grounding DINO with Prompt-Enriched Training](https://mlsanity.com/paper/2604.00503/) (2026)
- [DETR-ViP: Detection Transformer with Robust Discriminative Visual Prompts](https://mlsanity.com/paper/2604.14684/) (2026)
- [Expanding Zero-Shot Object Counting with Rich Prompts](https://mlsanity.com/paper/2505.15398/) (2025)
- [ChatGPT and general-purpose AI count fruits in pictures surprisingly well](https://mlsanity.com/paper/2404.08515/) (2024)
- [RWKV-UI: UI Understanding with Enhanced Perception and Reasoning](https://mlsanity.com/paper/2502.03971/) (2025)
- [ZERO: Industry-ready Vision Foundation Model with Multi-modal Prompts](https://mlsanity.com/paper/2507.04270/) (2025)
- [ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation](https://mlsanity.com/paper/2606.23835/) (2026)
- [Object Counting Across Modalities: Taxonomies, Benchmarks, Applications, and Open Challenges](https://mlsanity.com/paper/2608.23845/) (2026)
- [OPUS: A Simple yet Effective Unified Framework for Open-Vocabulary Detection](https://mlsanity.com/paper/2608.30247/) (2026)

## Abstract (arXiv)

We introduce T-Rex, an interactive object counting model designed to first detect and then count any objects. We formulate object counting as an open-set object detection task with the integration of visual prompts. Users can specify the objects of interest by marking points or boxes on a reference image, and T-Rex then detects all objects with a similar pattern. Guided by the visual feedback from T-Rex, users can also interactively refine the counting results by prompting on missing or falsely-detected objects. T-Rex has achieved state-of-the-art performance on several class-agnostic counting benchmarks. To further exploit its potential, we established a new counting benchmark encompassing diverse scenarios and challenges. Both quantitative and qualitative results show that T-Rex possesses exceptional zero-shot counting capabilities. We also present various practical application scenarios for T-Rex, illustrating its potential in the realm of visual prompting.