# Beyond Fixed Grid: Learning Geometric Image Representation with a Deformable Grid

arXiv: https://arxiv.org/abs/2008.09269 · HTML: https://mlsanity.com/paper/2008.09269/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2020.

## Built on by (most cited first, 16 of 16)

- [Deep Marching Tetrahedra: a Hybrid Representation for High-Resolution 3D Shape Synthesis](https://mlsanity.com/paper/2111.04276/) (2021)
- [GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images](https://mlsanity.com/paper/2209.11163/) (2022)
- [Adaptive Shells for Efficient Neural Radiance Field Rendering](https://mlsanity.com/paper/2311.10091/) (2023)
- [Coarse-Fine Networks for Temporal Activity Detection in Videos](https://mlsanity.com/paper/2103.01302/) (2021)
- [Continuous Memory Representation for Anomaly Detection](https://mlsanity.com/paper/2402.18293/) (2024)
- [SALISA: Saliency-based Input Sampling for Efficient Video Object Detection](https://mlsanity.com/paper/2204.02397/) (2022)
- [Learning Geometry-aware Representations by Sketching](https://mlsanity.com/paper/2304.08204/) (2023)
- [The Neurally-Guided Shape Parser: Grammar-based Labeling of 3D Shape Regions with Approximate Inference](https://mlsanity.com/paper/2106.12026/) (2021)
- [Palmira: A Deep Deformable Network for Instance Segmentation of Dense and Uneven Layouts in Handwritten Manuscripts](https://mlsanity.com/paper/2108.09436/) (2021)
- [Split GCN: Effective Interactive Annotation for Segmentation of Disconnected Instance](https://mlsanity.com/paper/2112.06454/) (2021)
- [Heterogeneous Grid Convolution for Adaptive, Efficient, and Controllable Computation](https://mlsanity.com/paper/2104.11176/) (2021)
- [Accelerating the creation of instance segmentation training sets through bounding box annotation](https://mlsanity.com/paper/2205.11563/) (2022)
- [Few-shot 3D Shape Generation](https://mlsanity.com/paper/2305.11664/) (2023)
- [Unit Region Encoding: A Unified and Compact Geometry-aware Representation for Floorplan Applications](https://mlsanity.com/paper/2501.11097/) (2025)
- [Teaching AI the Anatomy Behind the Scan: Addressing Anatomical Flaws in Medical Image Segmentation with Learnable Prior](https://mlsanity.com/paper/2403.18878/) (2024)
- [Bi-Grid Reconstruction for Image Anomaly Detection](https://mlsanity.com/paper/2504.00609/) (2025)

## Abstract (arXiv)

In modern computer vision, images are typically represented as a fixed uniform grid with some stride and processed via a deep convolutional neural network. We argue that deforming the grid to better align with the high-frequency image content is a more effective strategy. We introduce Deformable Grid DefGrid, a learnable neural network module that predicts location offsets of vertices of a 2-dimensional triangular grid, such that the edges of the deformed grid align with image boundaries. We showcase our DefGrid in a variety of use cases, i.e., by inserting it as a module at various levels of processing. We utilize DefGrid as an end-to-end learnable geometric downsampling layer that replaces standard pooling methods for reducing feature resolution when feeding images into a deep CNN. We show significantly improved results at the same grid resolution compared to using CNNs on uniform grids for the task of semantic segmentation. We also utilize DefGrid at the output layers for the task of object mask annotation, and show that reasoning about object boundaries on our predicted polygonal grid leads to more accurate results over existing pixel-wise and curve-based approaches. We finally showcase DefGrid as a standalone module for unsupervised image partitioning, showing superior performance over existing approaches. Project website: http://www.cs.toronto.edu/~jungao/def-grid