Partial convolution based multimodal autoencoder for art investigation

Autoencoders have been widely used in applications with limited annotations to extract features in an unsupervised manner, pre-processing the data to be used in machine learning models. This is especially helpful in image processing for art investigation where annotated data is scarce and difficult to collect. We introduce a structural similarity index based loss function to train the autoencoder for image data. By extending the recently developed partial convolution to partial deconvolution, we construct a fully partial convolutional autoencoder (FP-CAE) and adapt it to multimodal data, typically utilized in art investigation. Experimental results on images of the Ghent Altarpieceshow that our method significantly suppresses edge artifacts and improves the overall reconstruction performance. The proposed FP-CAE can be used for data preprocessing in craquelure detection and other art investigation tasks in future studies.

Jaar van publicatie:2019

Toegankelijkheid:Open