mlsanityData to 7 Oct 2026

The Functional Correspondence Problem

arXiv 2109.01097 ↗First posted 2 Sep 2021cs.CV cs.LG cs.ROcited by 20

Not on the map. Why?

Abstract (arXiv)

The ability to find correspondences in visual data is the essence of most computer vision tasks. But what are the right correspondences? The task of visual correspondence is well defined for two different images of same object instance. In case of two images of objects belonging to same category, visual correspondence is reasonably well-defined in most cases. But what about correspondence between two objects of completely different category -- e.g., a shoe and a bottle? Does there exist any correspondence? Inspired by humans' ability to: (a) generalize beyond semantic categories and; (b) infer functional affordances, we introduce the problem of functional correspondences in this paper. Given images of two objects, we ask a simple question: what is the set of correspondences between these two images for a given task? For example, what are the correspondences between a bottle and shoe for the task of pounding or the task of pouring. We introduce a new dataset: FunKPoint that has ground truth correspondences for 10 tasks and 20 object categories. We also introduce a modular task-driven representation for attacking this problem and demonstrate that our learned representation is effective for this task. But most importantly, because our supervision signal is not bound by semantics, we show that our learned representation can generalize better on few-shot classification problem. We hope this paper will inspire our community to think beyond semantics and focus more on cross-category generalization and learning representations for robotics tasks.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 32 of its 61 references that are in the map. First posted on arXiv 2 Sep 2021.

Builds on

32 of 61 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 2016FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networkscited by 780
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2019Your Classifier is Secretly an Energy Based Model and You Should Treat it Like Onecited by 402
  9. 2017End-to-End Learning of Geometry and Context for Deep Stereo Regressioncited by 359
  10. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  11. 2020Space-Time Correspondence as a Contrastive Random Walkcited by 239
  12. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
20 more
  1. 2017Convolutional neural network architecture for geometric matchingcited by 138
  2. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  3. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  4. 2019SelFlow: Self-Supervised Learning of Optical Flowcited by 128
  5. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  6. 2016Learning Modular Neural Network Policies for Multi-Task and Multi-Robot Transfercited by 124
  7. 2019Task-Driven Modular Networks for Zero-Shot Compositional Learningcited by 121
  8. 2020MAST: A Memory-Augmented Self-supervised Trackercited by 109
  9. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  10. 2015EpicFlow: Edge-Preserving Interpolation of Correspondences for Optical Flowcited by 80
  11. 2017Accurate Optical Flow via Direct Cost Volume Processingcited by 66
  12. 2017End-to-end weakly-supervised semantic alignmentcited by 60
  13. 2017FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondencecited by 42
  14. 2018Binge Watching: Scaling Affordance Learning from Sitcomscited by 41
  15. 2016Learning to Push by Grasping: Using multiple tasks for effective learningcited by 36
  16. 2019TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learningcited by 32
  17. 2020Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Graspingcited by 26
  18. 2019Self-supervised Learning for Video Correspondence Flowcited by 21
  19. 2017AnchorNet: A Weakly Supervised Network to Learn Geometry-sensitive Features For Semantic Matchingcited by 13
  20. 2015In Defense of the Direct Perception of Affordancescited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 0; 2023: 3; 2024: 5; 2025: 9; 2026: 1.

20 papers.

Sort:

By area:

  1. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  2. 2021Human Hands as Probes for Interactive Object Understandingcited by 51
  3. 2023Context Understanding in Computer Vision: A Surveycited by 23
  4. 2024UniGarmentManip: A Unified Framework for Category-Level Garment Manipulation via Dense Visual Correspondencecited by 22
  5. 2024Keypoint Abstraction using Large Models for Object-Relative Imitation Learningcited by 18
  6. 2025CordViP: Correspondence-based Visuomotor Policy for Dexterous Manipulation in Real-Worldcited by 16
  7. 2025Do It Yourself: Learning Semantic Correspondence from Pseudo-Labelscited by 13
  8. 2025Visual Intention Grounding for Egocentric Assistantscited by 11
  9. 2023STRAP: Structured Object Affordance Segmentation with Point Supervisioncited by 10
  10. 2021IFR-Explore: Learning Inter-object Functional Relationships in 3D Indoor Scenescited by 9
  11. 2023Learning Reusable Manipulation Strategiescited by 8
  12. 2025FUNCTO: Function-Centric One-Shot Imitation Learning for Tool Manipulationcited by 8
8 more
  1. 2024One-Shot Manipulation Strategy Learning by Making Contact Analogiescited by 6
  2. 2025Adapting by Analogy: OOD Generalization of Visuomotor Policies via Functional Correspondencecited by 3
  3. 2025Towards Geometry-Grounded Dense Semantic Matching with VGGT Priorscited by 3
  4. 2026MARCO: Navigating the Unseen Space of Semantic Correspondencecited by 3
  5. 2025Weakly-Supervised Learning of Dense Functional Correspondencescited by 2
  6. 2025Understanding Task Transfer in Vision-Language Modelscited by 2
  7. 2024Learning secondary tool affordances of human partners using iCub robot's egocentric datacited by 0
  8. 2025A Recursive Pyramidal Algorithm for Solving the Image Registration Problemcited by 0