This is a master of science thesis by Doğukan Erenel and advisor Assoc. Prof. Haluk Bingöl.

---

This work is partially supported by Boğaziçi University Research Fund Project BAP-2011-6017.

---

Within the popularity of new interface devices such as accelerometer based game controllers or touch-screen smartphones, the need of new accessibility options for these interfaces have become emergent. Previous studies gave the idea of using accelerometer based gesture recognition system on touch-screen smartphones with accelerometer as a new interface for visually-impaired people to use touchscreen keyboards. However, almost all studies, which have high accuracy results, are used user-dependent classifications or very limited gesture sets.

In this study, our aim is to find an alternative approach to classify 20 different gestures captured by iPhone 3GS’s built-in accelerometer and make high accuracy on user-independent classifications. Our method based on Dynamic Time Warping (DTW) with dynamic warping window sizes. The first experimental result, which is obtained from collected data set, gives 96.7% accuracy rate among 20 gestures with 1062 gesture data totally. The second experimental result, which is obtained from 4 visually-impaired people with implemented calculator as end-user test, gives 95.5% accuracy rate among 15 gestures with 720 gesture data totally. Within this work, a design of accelerometer-based recognition system is given as well as its implementation as a gesture recognition based talking calculator and experimental results.

See Demo Video : http://vimeo.com/26196932 Or http://www.youtube.com/watch?v=VPzg3gHINPs

---

Data Set (which is explained below) (.plist): Download Data Set

Training results (which is explained below) (.plist): Download Training Result

Gesture Recognition Framework (made with objective-c and XCode) used in both training and classification parts (.framework): Download Framework

Gesture Recognition Project : Download XCode Project

Thesis Document (.pdf) : Thesis in PDF

Article Document (.pdf) : available soon…

---

Image	Name	Info	Position
	Doğukan Erenel	Email: ereneld at gmail.com Web: LinkedIn Address: ETA 21, Bogazici University; 34342 Bebek, Istanbul; Turkey	Author
	Haluk Bingöl Dept. of Computer Engineering Bogazici University	Tel: +90-(212)-359-7121 Email: bingol at boun.edu.tr Address: ETA 48; Bogazici University; 34342 Bebek, Istanbul; Turkey	Advisor

---

There are 1091 gesture data (grouped according to actual class number) in one XML file with given information;

1. Age
2. Profession (0 : No Job)
   1. Unknown Job
   2. Student
   3. Teacher
   4. Worker
   5. Doctor
   6. Engineer
   7. Sport Profession
   8. Art Profession
   9. Other
3. Education (0 : No Education)
   1. Unknown Education
   2. Read Write
   3. Elementary School
   4. High-School
   5. UnderGraduate
   6. Master Degree
   7. Phd Degree
   8. Professor
4. Disability (Y: Yes or N:No)
5. Sex (M: Male, F: Female, ?)
6. Hand usage (L: Left, R: Right)
7. Current hand usage (L: Left, R: Right)
8. 3D Accelerometer data for given gesture type with 4 data sequences respectively;
   1. Timestamp for each accelerometer update clock
   2. Gesture data sequence in X axis (time series in x axis)
   3. Gesture data sequence in Y axis (time series in y axis)
   4. Gesture data sequence in Z axis (time series in z axis)
9. Date of data collected

Collected gestures are given below;

1), 2), 3), 4)

5), 6), 7), 8)

9), 10), 11), 12)

13), 14), 15), 16)

17), 18), 19), 20)

21), 22)

Gestures used in training and calculator are given below;

---

In this study; two experiments were made for validating our recognition system. In the first experiment; training set were collected from 15 participant, who do not have any disability, with totally 1091 gesture data. However, in the data validation step the system eliminated 29 gesture data, so 1062 gesture data were used totally. After data collection our system was trained and validated with leave-one-out cross validation. For the second experiment, a talking calculator was made, which uses the implemented framework. Here, Flite text-to-speech library and its wrapper by Sam Foster was used. In order to test our system with the calculator; 40 calculations, containing totally 180 characters, were defined. Then 4 visually-impaired people tested the system by making defined calculations using our calculator in 7 days, here the number of days is determined intuitively. In each day, all participants calculation data were collected in order to check the learning curve and the accuracy of the classifications.

Second experiment and result:

---

---

Triangle Motion (dynamic camera) - Accelerometer Simulation : http://vimeo.com/25746547

Circle Motion (static camera) - Accelerometer Simulation : http://vimeo.com/25743050

Triangle Motion (static camera) - Accelerometer Simulation : http://vimeo.com/25626775

Circle Motion (dynamic camera) - Accelerometer Simulation : http://vimeo.com/25623138