Specifying Gestures by Example

The paper discusses GRANDMA(Gesture Recognizers Automated in Novel Direct Manipulation Architecture) and GDP(Gesture-based drawing program) which was built using GRANDMA. With GRANDMA one can create gesture recognizers from example gestures, removing the need for hand-coding. In already existing gesture-based applications, the recognizer was hand coded which makes it difficult to create, maintain and modify.

GDP: The user begins the gesture by placing the mouse cursor and pressing the mouse button. The the user draws by moving the mouse. The end of gesture can be either by just releasing the mouse button in which case the gesture is recognized and drawn or can stop moving the mouse for given time  while pressing the button in which case the  recognized gesture is drawn and it allows dragging to resize and reshape the recognized shape. This is two-phase interaction, gesture collection followed by classification and manipulation. The gestures in GDP are all single strokes. This avoids segmentation problem which would be present with multi-stroke character recogniton and allows shorter timeouts to be used.\

GRANDMA is object oriented toolkit used to build click-and-drag interface and the application.  The following is the GDP's the view class hierarchy.

It is MVC like system. A single input handler may be associated with a view class and shared between instances of the class. Using GRANDMA one can easily add new gestures and give training samples for it. The semantic of each gesture can be changed by entering expressions for each of recog(which is evaluated when gesture is recognized), manip(which is evaluated on subsequent mouse points) and done(which is evaluated when mouse button is released)

Gesture recognition: A vector of features is extracted from input and it is classified as one of the C possible gestures via a linear machine. The features for this tool should be chosen considering following:

Small change in input should result in small change in each feature.

There should be enough features to differentiate gestures but there should not be too many that it starts affecting efficiency

Feature should be incrementally computable in constant time per input point.

Gesture classification : Gesture class c has weights wci for 0<i<=F where F is the total number of features. The classification of gesture is the one which maximizes the vc.

Training: Training is to determine wci from example gestures. A well-known closed formula was used.

Rejection: Ambiguous gestures and outliers should be rejected. Gestures having P(g was classified correctly)<0.95 are rejected. Another method uses Mahalanobis distance to determine standard deviation and rejects gestures having (standard deviation)^2 > (1/2)F^2. Rejection should be disabled in applications since it increases number of gestures that need to be redone. 

Thoughts:

User should be allowed to define his own features.

The tool can extend to include animation gestures too.

Different kinds of interfacing like finger touch  can be used.

Who Dotted the 'i'?

This paper describes a system which can automatically determine a stroke's owner in a collaborative environment even when strokes are drawn using the same pen, in close proximity and in close timing. It identifies user purely based on physical mannerisms of how the strokes are made. The mannerisms are tilts of the pen, pressure of the pen and the speed at which the pen moves.

Unobtrusive user differentiation can be used for security and verification, improved sketch recognition through user-modelling recognition and collaboration.

Two experiments were conducted using a drawing application created in Cocoa for Mac OS X and Cintiq which was used to collect data such as tilt and pressure of the pen. First experiment had six participants and its purpose was to determine if certain features could be used to disambiguate users and how to disambiguate using tilt and pressure. The result showed that users are fairly consistent with physical manner in which they sketch and their mannerisms are distinct from one another. Experiment two was conducted using 10 participants to determine if a classifier could be created that can accurately determine creator of a stroke from set of possible creators. The classifier used 24 features and a variety of learners like Linear classifier, Quadratic classifier. K nearest neighbor gave the best results. The influence of size of the creator set on accuracy to determine the creator was also tested. The identification rate was 97.5% and 83.5% for two and ten users, respectively.

By using additional context the accuracy can be improved.

Mechanix

Authors: Stephanie Valentine, Francisco Vides, George Lucchese, David Turner,
Hong-hoe Kim, Wenzhe Li, Julie Linsey, Tracy Hammond

Citation:http://www.aaai.org/ojs/index.php/aimagazine/article/view/2437

This paper describes about Mechanix which is a sketch-based tutoring and grading system for free-body diagram. It can recognize, correct and provide real-time feedback on a student's sketch that is compared with the instructor-entered key sketch. It is capable of understanding the student sketches and solve students' trusses without an instructor-provided sketch.

There are two modes supported by Mechanix - Student mode and Instructor mode. The interface of student mode contains text at the top center describing the problem, tool panel, checklist providing step-by-step guide to solve the problem, drawing panel, pullout notepad to make notes, equation panel. The system also has a feedback bar. The instructor mode allows instructors to add problem statement in form of simple text and explanatory images. They can also sketch a solution which the system uses to compare the student solutions. Instructors label nodes and forces and Mechanix generates equations and values for reaction forces. One can continually modify and edit their drawings using round menu, buttons and freehand erasure like on pen and paper.

Geometric Recognition is used in Mechanix. It uses bottom-up approach. The various steps involved are preprocessing, primitive recognition to identify basic shapes like lines, arcs or circles and  high-level recognition that uses constraints to determine complex shapes from the basic shapes. Mechanix uses cusp detector to identify strokes and a low-level recognizer called PaleoSketch which supports more than 10 basic shapes. Primitive recognition is also used to give visual feedback. For example, shape recognized as forced is changed to lime green indicating student may enter relevant meta data.

Mechanix supports truss diagram, non-truss diagram and creative response problems. Mechanix identifies trusses and compares with the instructor's values to give appropriate feedback. For non-truss body diagrams, a generic closed-shape comparison technique is used to compare student solutions with that of instructor. The other kind of problem that Mechanix supports is Creative response. Here it uses artificial intelligence to check the student-created truss. It creates linear system of equation and a system of equation is formed for each beam in truss. The Mechanix will use the values to compare to a set of constraints that the instructor previously entered and provides feedback.

The system keeps track of log of mistakes that a student commits while solving the problem. The answers are checked in a server to prevent cheating and feedback is sent to the client installed on student device. XML over HTTP is used to transfer data between server and client.


Thoughts:
The idea of having an application that allows students to solve assignments, gives continuous feedback and corrects the solution submitted is very helpful to both students and instructors. It would motivate the students to solve the assignments and instructors don't have to waste time in correcting each student's solution.

The most impressive feature is freehand erasure. The interface is similar to what one would do in real life when one makes a mistake while drawing and has no patience to use eraser. Other idea that I think are pretty cool are that of visual feedback(changing of the color of the force to get student's input value of force).