Sign Language Tutoring Tool


This project aims sign language training of the deaf and mute people in an interactive environment. Instead of a passive learner, by incorporating a vision-based sign language recognizer that uses simple and easy-to-find webcams, the learner would be able to practice the signs and have feedbacks according to the similarity of the performed gesture to the actual gesture model.

During the learning process of sign language, one of the most important issues is to validate if the learned sign is correct or not. Instead of asking someone who knows sign language, this validation can be done by this demonstrator. This interactive sign language teaching application can be used by deaf and mute people as well as by users with no disability. The availability of this application will increase the number of people who speak sign language and as a result, the amount of communication among deaf and mute people and people with no disability.

This Sign Language Tutoring Tool is an interactive teaching platform where the users watch pre-recorded signs from natural signers to learn that sign. The performed sign of the user is captured by the program, processed according to signer’s hand motion, general hand shape (and hand position) and then compared to the gestures in the database. A feedback is given to the user about the quality of the performance. Kalman Filter is used for hand tracking and left-to-right HMMs are used for recognition of the gestures.

There are currently 7 signs from Turkish Sign Language in the system and the user selects among these different signs. First a video of the sign, which is performed by a natural signer, is shown to the learner and then the learner is asked to perform the sign in the video. The two gestures (performed and actual) are compared and a feedback is given to the user as TRUE or FALSE.

The signs are determined such that they contain the movement of the hand.

7 TSL Signs used in the system

The user interface is a dialog based MFC application and designed such that the user can select and practice different signs. The interface form is divided into two sections which reflects the two phases of the system: Learning phase and practice phase. In the learning phase, the user first selects a sign. A pre-recorded video of that sign is shown when the user presses play button. The user can repeatedly watch this video until she/he is ready to practice. In the practice phase, the user is asked to repeat the same gesture which is selected in the training phase. Recording continues until the hand is out of sight, or until the maximum duration is reached. Maximum duration for recording is determined according to the duration of the training video. At the end of the recording, the recorded file is processed and feedback is given to the user as TRUE/FALSE. The user can practice the same sign repeatedly. The training phase can be done for each of the sign in the application library.

Application Requirements

  • This demonstrator is designed to run on Windows XP operating system and implemented using Microsoft Visual Studio C++ 6.0.
  • Pentium 3 1.6 GHz PC will be enough for real time operation.
  • 2 web cameras that are capable of sending uncompressed RGB bitmap images must be installed. The bandwidths of the USB ports of two web cameras must be enough for the transmission of RGB bitmap images. (We have used Creative web cams with 30 frames/sec)
  • A single colored glove (Color of the glove must be different than the colors of the objects in the view of the cameras)

    Download

    Download (only one sign video is included) -- 11MB
    Download and unzip to any folder. Run FingerTracker.exe after connecting web cams. Set the resolution and fps of both cameras to the same values.
    Video of the demonstrator -- 6.6MB
    In this video, an example for the use of Sign Language Tutor is shown. The user first registers the marker (glove) to the system. Then the user watchs pre-recorded videos of some of the signs (anne, cuma, gormek). Third, the user practices two of the signs: gormek and kolay.

    O. Aran, C. Keskin, L. Akarun, "Sign Language Tutoring Tool", EUSIPCO’05, Antalya, Turkey, Sept. 4-8, 2005.