Spatially Grounded Concept Bottleneck Models via Part-Factorized Attention

Concept bottleneck models (CBMs) predict a layer of human-named attributes before predicting a class, which makes their decisions auditable. On fine-grained recognition tasks, though, the concept heads are usually free to attend anywhere in the image, so a head named for one body region can be satisfied by evidence on another, and the model reaches the right answer for the wrong reason. We propose a part-factorized CBM (PF-CBM) that removes this freedom by construction. A frozen DINOv3 vision transformer feeds a set of part queries, each tied by name to a specific anatomical region through a fixed concept-to-part map, while whole-object attributes such as size and shape are handled separately by a query with no spatial prior, since they are not anchored to any single body part. A learnable Gaussian prior over patch locations, initialized from average keypoint positions, keeps the part queries from collapsing onto the same evidence. On its own this prior spreads the queries apart but does not reliably land them on the correct anatomy. What closes that gap is a lightweight alignment loss that nudges each part query toward its keypoint, and the central finding of this paper is how little of that supervision is required. Aligning on well under one percent of the training images already moves pointing accuracy from near-chance to roughly three-quarters of what full keypoint supervision achieves, and the gains continue, more slowly, as more annotated images are added. Classification accuracy on CUB-200-2011 barely moves across this entire range and stays within a point of a fully supervised baseline whether the model sees no keypoints at all or every one of them. Grounding a CBM's attention to the right evidence turns out to be nearly free in accuracy and cheap in annotation, provided the model has the right inductive bias to make efficient use of that small amount of supervision.