< Back to previous page

Publication

Multi-modal label retrieval for the visual arts

Book Contribution - Book Abstract Conference Contribution

Subtitle:the case of Iconclass
Iconclass is an iconographic classification system from the domain of cultural heritage which is used to annotate subjects represented in the visual arts. In this work, we investigate the feasibility of automatically assigning Iconclass codes to visual artworks using a cross-modal retrieval set-up. We explore the text and image branches of the cross-modal network. In addition, we describe a multi-modal architecture that can jointly capitalize on multiple feature sources: textual features, coming from the titles for these artworks (in multiple languages) and visual features, extracted from photographic reproductions of the artworks. We utilize Iconclass definitions in English as matching labels. We evaluate our approach on a publicly available dataset of artworks (containing English and Dutch titles). Our results demonstrate that, in isolation, textual features strongly outperform visual features, although visual features can still offer a useful complement to purely linguistic features. Moreover, we show the cross-lingual (Dutch-English) strategy to be on par with the monolingual approach (English-English), which opens important perspectives for applications of this approach beyond resource-rich languages.
Book: Proceedings of the 13th International Conference on Agents and Artificial Intelligence (ICAART 2021) - Volume 1
Pages: 622 - 629
ISBN:978-989-758-484-8
Publication year:2021
Keywords:P1 Proceeding
Accessibility:Open