| Mutual Disambiguation of Verbal and Sketching Inputs in a Design Environment (2008) | |||||||||||||||
Abstract | |||||||||||||||
| The Problem: In any design environment, some things are more easily expressed verbally, while some are more easily expressed visually in a sketch. We would like to provide a design environment where the interaction is as natural as possible, and thus want to have both forms of input. In addition to making the interaction more natural, having speech recognition will allow mutual disambiguation between the sketch and the speech. Motivation: Our previous sketching system [1] recognizes sketches of mechanical systems drawn with a pen-like stylus. However, we soon noticed that some things are difficult to express visually. For example, it is easy to say that we want three identical equally spaced objects, but far more difficult to draw this reliably. Also, speech input can supply a second source of information that can be used to disambiguate the sketch. This will create a natural and easy-to-use environment where the computer can assist the user without being obtrusive. Figure 1: A user sketching and speaking to the system Previous Work: Other work on sketching and multi-modal systems includes systems like QuickSet [5]. There has been research done on speech and sketching integration in QuickSet [3, 2]. Our work is different in that we want the interaction to be more than simply individual commands of the form ”place a bridge here. ” We want to deal with speech that is not necessarily grammatical, which makes speech recognition a much harder task. The goal of our system is to obtain what information it can from the user’s natural speech and use that to disambiguate the sketch. | |||||||||||||||
Publication details | |||||||||||||||
| |||||||||||||||