This project aims sign language training of the deaf and mute people in an interactive environment. Instead of a passive learner, by incorporating a vision-based sign language recognizer that uses simple and easy-to-find webcams, the learner would be able to practice the signs and have feedbacks according to the similarity of the performed gesture to the actual gesture model.
During the learning process of sign language, one of the most important issues is to validate if the learned sign is correct or not. Instead of asking someone who knows sign language, this validation can be done by this demonstrator. This interactive sign language teaching application can be used by deaf and mute people as well as by users with no disability. The availability of this application will increase the number of people who speak sign language and as a result, the amount of communication among deaf and mute people and people with no disability.
This Sign Language Tutoring Tool is an interactive teaching platform where the users watch pre-recorded signs from natural signers to learn that sign. The performed sign of the user is captured by the program, processed according to signer’s hand motion, general hand shape (and hand position) and then compared to the gestures in the database. A feedback is given to the user about the quality of the performance. Kalman Filter is used for hand tracking and left-to-right HMMs are used for recognition of the gestures.
There are currently 7 signs from Turkish Sign Language in the system and the user selects among these different signs. First a video of the sign, which is performed by a natural signer, is shown to the learner and then the learner is asked to perform the sign in the video. The two gestures (performed and actual) are compared and a feedback is given to the user as TRUE or FALSE.
The signs are determined such that they contain the movement of the hand.
|7 TSL Signs used in the system|
The user interface is a dialog based MFC application and designed such that the user can select and practice different signs. The interface form is divided into two sections which reflects the two phases of the system: Learning phase and practice phase. In the learning phase, the user first selects a sign. A pre-recorded video of that sign is shown when the user presses play button. The user can repeatedly watch this video until she/he is ready to practice. In the practice phase, the user is asked to repeat the same gesture which is selected in the training phase. Recording continues until the hand is out of sight, or until the maximum duration is reached. Maximum duration for recording is determined according to the duration of the training video. At the end of the recording, the recorded file is processed and feedback is given to the user as TRUE/FALSE. The user can practice the same sign repeatedly. The training phase can be done for each of the sign in the application library.
O. Aran, C. Keskin, L. Akarun, "Sign Language Tutoring Tool", EUSIPCO’05, Antalya, Turkey, Sept. 4-8, 2005.