Towards a composite event-based language for describing multimodal interactions

Implementing interactive systems with event languages requires writing subroutines, called event handlers, which, at runtime, are automatically called when external events occur. When implementing multimodal systems with event languages, the interaction code gets split into several event handlers within which a multitude of flags and state variables have to be manually maintained in a self-consistent manner, thus complicating programmers' work. The present research aims at simplifying this complexity with a language that empowers programmers to define event sequences, herein called composite events, each of which can be bound to one or more event handlers. At runtime, these event handlers will be called automatically at different stages of the composite event detection process. Hasselt, the proposed language, comes accompanied with a supporting tool that includes the editors, compilers, runtime environment, and debugging tools required to write, syntax-check, run, and test Hasselt programs, respectively. Using an event language as a baseline, Hasselt was evaluated by both static and dynamic testing. First, code inspection was used to evaluate the complexity of equivalent source codes written with both Hasselt and an event language. Among many results, the inspection showed that Hasselt code is shorter and simpler since it releases programmers from manually tracking sequences of events. Second, a user study was conducted to compare programming efficiency. After modifying an interaction model with both Hasselt and the baseline language, it was revealed that the former leads to higher completion rates, lower completion times, and less code testing. The results obtained during this research imply that augmenting existing event languages with notations for defining composing events may be one way to reduce the accidental complexity of implementing multimodal systems.

Publication year:2016

Keywords:event languages, composite events, multimodal interaction, interaction model

Accessibility:Open