Pictographic Communication Technologies for Browsing the Web.

In order to improve the accessibility of the Internet for users with reading and writing disabilities, we develop a set of tools that automatically translate Dutch natural language text into pictographs and vice versa for people with an intellectual disability (ID), allowing them to read and write status updates, emails, and chat messages in online environments.

For the conversion of texts into pictographs, we start from an existing system (Vandeghinste et al. 2017).  We evaluate the baseline Text-to-Pictograph translation system using automated metrics, manual assessments, and focus groups with real end users, and propose three improvements: We create a spelling correction tool for people with ID, we develop a syntactic simplification tool and a temporality detection module that uses deep syntactic analysis, and we implement a word sense disambiguation tool for improved semantic analysis. The added value of each one of these components is measured by a combination of automated metrics, manual evaluations, and, where possible, user studies.

Conversely, the Pictograph-to-Text translation tool provides help in constructing Dutch textual messages by allowing a user to input a series of pictographs, and then translates these messages into natural language text. The challenge in Pictograph-to-Text translation is twofold. The first task involves the development of an accessible interface that allows people with ID to find the pictographs of their choice. The second task concerns the actual development of the Pictograph-to-Text translation engine. We discuss a variety of approaches, including language modelling and (neural) machine translation techniques, toward the generation of rich natural language text from underspecified pictograph input.