# Interpretable Transformations with Encoder-Decoder Networks

arXiv: https://arxiv.org/abs/1710.07307 · HTML: https://mlsanity.com/paper/1710.07307/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Oct 2017.

## Built on by (most cited first, 30 of 34)

- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [Vector Neurons: A General Framework for SO(3)-Equivariant Networks](https://mlsanity.com/paper/2104.12229/) (2021)
- [From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI](https://mlsanity.com/paper/2201.08164/) (2022)
- [3D Neural Scene Representations for Visuomotor Control](https://mlsanity.com/paper/2107.04004/) (2021)
- [GTA: A Geometry-Aware Attention Mechanism for Multi-View Transformers](https://mlsanity.com/paper/2310.10375/) (2023)
- [UmeTrack: Unified multi-view end-to-end hand tracking for VR](https://mlsanity.com/paper/2211.00099/) (2022)
- [Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models](https://mlsanity.com/paper/2209.06970/) (2022)
- [Explainable AI for Time Series via Virtual Inspection Layers](https://mlsanity.com/paper/2303.06365/) (2023)
- [Steerers: A framework for rotation equivariant keypoint descriptors](https://mlsanity.com/paper/2312.02152/) (2023)
- [Equivariant Representation Learning via Class-Pose Decomposition](https://mlsanity.com/paper/2207.03116/) (2022)
- [SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learning](https://mlsanity.com/paper/2301.11520/) (2023)
- [Learning Generalizable Light Field Networks from Few Images](https://mlsanity.com/paper/2207.11757/) (2022)
- [Unsupervised Novel View Synthesis from a Single Image](https://mlsanity.com/paper/2102.03285/) (2021)
- [Towards Self-Supervised Gaze Estimation](https://mlsanity.com/paper/2203.10974/) (2022)
- [Query-based Semantic Gaussian Field for Scene Representation in Reinforcement Learning](https://mlsanity.com/paper/2406.02370/) (2024)
- [Multi-view Hand Reconstruction with a Point-Embedded Transformer](https://mlsanity.com/paper/2408.10581/) (2024)
- [Semi-supervised Viewpoint Estimation with Geometry-aware Conditional Generation](https://mlsanity.com/paper/2104.01103/) (2021)
- [CUDA-GHR: Controllable Unsupervised Domain Adaptation for Gaze and Head Redirection](https://mlsanity.com/paper/2106.10852/) (2021)
- [Efficient View Synthesis and 3D-based Multi-Frame Denoising with Multiplane Feature Representations](https://mlsanity.com/paper/2303.18139/) (2023)
- [Learning Neural Representation of Camera Pose with Matrix Representation of Pose Shift via View Synthesis](https://mlsanity.com/paper/2104.01508/) (2021)
- [NeReF: Neural Refractive Field for Fluid Surface Reconstruction and Implicit Representation](https://mlsanity.com/paper/2203.04130/) (2022)
- [Modular Machine Learning: An Indispensable Path towards New-Generation Large Language Models](https://mlsanity.com/paper/2504.20020/) (2025)
- [Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth](https://mlsanity.com/paper/2107.13269/) (2021)
- [ViewNet: Unsupervised Viewpoint Estimation from Conditional Generation](https://mlsanity.com/paper/2212.00435/) (2022)
- [Disentangling Geometric Deformation Spaces in Generative Latent Shape Models](https://mlsanity.com/paper/2103.00142/) (2021)
- [DifAttack++: Query-Efficient Black-Box Adversarial Attack via Hierarchical Disentangled Feature Space in Cross-Domain](https://mlsanity.com/paper/2406.03017/) (2024)
- [Human Pose Manipulation and Novel View Synthesis using Differentiable Rendering](https://mlsanity.com/paper/2111.12731/) (2021)
- [Self-Supervised Equivariant Scene Synthesis from Video](https://mlsanity.com/paper/2102.00863/) (2021)
- [Transformation Coding: Simple Objectives for Equivariant Representations](https://mlsanity.com/paper/2202.10930/) (2022)
- [Equivariant Representation Learning in the Presence of Stabilizers](https://mlsanity.com/paper/2301.05231/) (2023)

## Abstract (arXiv)

Deep feature spaces have the capacity to encode complex transformations of their input data. However, understanding the relative feature-space relationship between two transformed encoded images is difficult. For instance, what is the relative feature space relationship between two rotated images? What is decoded when we interpolate in feature space? Ideally, we want to disentangle confounding factors, such as pose, appearance, and illumination, from object identity. Disentangling these is difficult because they interact in very nonlinear ways. We propose a simple method to construct a deep feature space, with explicitly disentangled representations of several known transformations. A person or algorithm can then manipulate the disentangled representation, for example, to re-render an image with explicit control over parameterized degrees of freedom. The feature space is constructed using a transforming encoder-decoder network with a custom feature transform layer, acting on the hidden representations. We demonstrate the advantages of explicit disentangling on a variety of datasets and transformations, and as an aid for traditional tasks, such as classification.