This article is cited in 6 papers

Cognitive study of an assistive multimodal user interface for hands-free human–computer interaction

A. A. Karpov

St. Petersburg Institute for Informatics and Automation of RAS (SPIIRAS)

Abstract: The process of research and development of a multimodal user interfaces aimed at hands-free interaction with a personal computer by means of the natural speech input and pointing gestures/movements by user's head is described. The proposed interface uses a low-cost audio-visual equipment for information input and provides an universal access to computer systems both for regular human-operators for contactless (without using hands) personal computer control, and for handicapped users having difficulties with hand motion or even without arms. The methods and results of quantitative evaluation of speech and performance and contactless human–computer interaction are described and comparison with contact-based means of information input is made.

Keywords: multimodal user interface; automatic speech recognition; computer vision; cognitive research.