MAChine Communication with Humans and Inference in NAtural environments (MACCHINA)

The project focuses on the core fundamental problem of how autonomous systems and their users can naturally interact about the shared context that they all see. However, for such functioning in realistic and natural environments, we lack technology that enables accurate and real-time automated understanding of the situation at hand. Visual and language data offer here complementary information (e.g., in an order like `follow the red car immediately ahead’), making their joint processing indispensable. MACCHINA will leverage recent advances in visual and language processing where deep learning approaches provide a shared methodology of interpreting content with the help of continuous representations. Fundamental research is needed to learn multimodal continuous representations that jointly capture the meaning of language and the visual reality and that allow visual situations to be translated into language and language into visuals. Such research on its turn contributes to the explanation of the learned representations. To instantly process and generate the dialogue between a human and the machine about the shared, visual environment, the machine constantly needs to predict or anticipate possible future situations enabling proactive reaction (as humans do), for which we lack principled models. Finally, an autonomous machine reacts in a real spatial environment, so the project investigates how the continuous content representations contribute to fast computation and inference in such spaces.

Keywords:Computer vision, Natural language processing, Multimodal understanding, Representation learning, Machine learning

Disciplines:Applied mathematics in specific fields, Computer architecture and networks, Distributed computing, Information sciences, Information systems, Programming languages, Scientific computing, Theoretical computer science, Visual computing, Other information and computing sciences