Dense Multitask Learning to Reconfigure Comics
Not on the map. Why?
Abstract (arXiv)
In this paper, we develop a MultiTask Learning (MTL) model to achieve dense predictions for comics panels to, in turn, facilitate the transfer of comics from one publication channel to another by assisting authors in the task of reconfiguring their narratives. Our MTL method can successfully identify the semantic units as well as the embedded notion of 3D in comic panels. This is a significantly challenging problem because comics comprise disparate artistic styles, illustrations, layouts, and object scales that depend on the authors creative process. Typically, dense image-based prediction techniques require a large corpus of data. Finding an automated solution for dense prediction in the comics domain, therefore, becomes more difficult with the lack of ground-truth dense annotations for the comics images. To address these challenges, we develop the following solutions: 1) we leverage a commonly-used strategy known as unsupervised image-to-image translation, which allows us to utilize a large corpus of real-world annotations; 2) we utilize the results of the translations to develop our multitasking approach that is based on a vision transformer backbone and a domain transferable attention module; 3) we study the feasibility of integrating our MTL dense-prediction method with an existing retargeting method, thereby reconfiguring comics.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 29 of its 52 references that are in the map. First posted on arXiv 16 Jul 2023.
Builds on
29 of 52 references.
References · 29
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2014Conditional Generative Adversarial Netscited by 2,162
- 2017Decoupled Weight Decay Regularizationcited by 1,831
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
- 2020Pre-Trained Image Processing Transformercited by 893
- 2017A Survey on Multi-Task Learningcited by 840
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
- 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
17 more
- 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
- 2018Taskonomy: Disentangling Task Transfer Learningcited by 634
- 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
- 2018End-to-End Multi-Task Learning with Attentioncited by 534
- 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 375
- 2015Sketch-based Manga Retrieval using Manga109 Datasetcited by 329
- 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
- 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
- 2020Robust Learning Through Cross-Task Consistencycited by 105
- 2021TVT: Transferable Vision Transformer for Unsupervised Domain Adaptationcited by 81
- 2019Many Task Learning with Task Routingcited by 50
- 2022MulT: An End-to-End Multitask Learning Transformercited by 48
- 2019Towards Instance-level Image-to-Image Translationcited by 36
- 2019Cycle-IR: Deep Cyclic Image Retargetingcited by 14
- 2018Unsupervised learning for cross-domain medical image synthesis using deformation invariant cycle consistency networkscited by 10
- 2021Spatio-Temporal Multi-Task Learning Transformer for Joint Moving Object Detection and Segmentationcited by 9
- 2021Estimating Image Depth in the Comics Domaincited by 8
Built on by
Citing papers per year: 2024: 1.
1 paper.