Report

SOMM: Self Organizing Markov Map for Gesture Recognition G. Caridakis et al., Pattern Recognition, Vol. 31, pp. 52-59, 2010. Pattern Recognition 2010 Spring Seung-Hyun Lee Contents • Introduction • Related Work – Hidden Markov Models – Other Method • Proposed Method • Experiments • Conclusion 1 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Introduction • Gesture – A motion of the body that conveys information • In this paper – Focus on hand gestures 2 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Introduction • Taxonomy of gesture(McNeill, 1992) – – – – – Gesticulation Speech-linked Pantomime Emblems Sign Languages • Other (Kendon,1992) (Quek, 1994) 3 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Introduction • Taxonomy by functionality Gestures Definition Symbolic gestures gestures that, within each culture, have come to have a single meani ng. Deictic gestures types of gestures most generally seen in HCI and are the gestures of pointing to entities or direction. Iconic gestures gestures used to convey information about the size, spatial relations, actions, shape or orientation of the object of discourse display. Pantominic gestures gestures typically used to mimic an action, object or concept. 4 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Hidden Markov Model Related Work • Cogan(2006) – Discrete HMM which fuse hand shape and position • Hossain(2005) – Implicit/Explicit Temporal Information Encoded HMM – Discriminated attention and non-attention gestures • Mantyla(2000) – On mobile devices – Utilized SOM and HMM method • Starner(1998) – HMM based American Sign Language(ASL) recognition – Sentence level recognition is possible 5 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Other method Related Work • Black and Jepson(1998) – CONDitional dENSity propagATION (CONDENSATION) algorothm • Wong and Ciipolla(2006) – Sparse Bayesian classifier • Hong et al.(2000) – Finite State Machines(FSM) • Su(2000) – Fuzzy logic and rule-based approaches and hyper-rectangular composite Neural network(HRCNNs) • Juang and Ku(2005) – Fuzzified Takagi-Sugeno-Kang(TSK) type recurrent network • Yang et al.(2002) – Time Delay Neural network • Huang and Huang(1998) – 3D Hopfield Neural Network 6 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Overview Proposed Method • Modules – Image processing : detection an tracking of hands – SOM : quantization of hand location and direction – HMM : transition probability matrix 7 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Feature Extraction Proposed Method • Video based method – Creation of moving skin masks (Skin color area) – Tracking the centroid of the skin masks – Prior knowledge is required • It should indicate different body parts (Left, right hand, and head) • Environment – PC platform – OpenCV 8 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Proposed Method • Dataset • Gesture instances • Gesture instances 9 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Position Model Proposed Method • cf) SOM (1) continuous input space (2) discrete output space in the form of lattice (3) time-varying neighborhood function defined around winning neuron (4) decreasing learning rate parameter 10 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Position Model Proposed Method • Some based representation of hand position 11 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Direction Model Proposed Method • Additional information: Moving direction 12 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Generalized Median Proposed Method • Based on Levenshtein distance(edit distance) – Measuring the amount of difference between two sequences • Generalized median of data set Mj • Mean Levenstein distance between members 13 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Gesture Decoding Proposed Method • Position – Probability – Calculation of Ssom • First state: initial probability • From second state: transition probability – Unit u 14 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Gesture Decoding Proposed Method • Direction – Probability – Calculation of Sof – Unit u 15 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Gesture Decoding Proposed Method • Similarity measurement – Problem • Shorter gesture instances tend to gain an advantage by having less transitions and thus less probabilities multiplication – Measurement 16 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Error Propagation Proposed Method • Error definition for function f • SOM based approach – If data containing small error is mapped to the same node of SOM No problem – Otherwise Consequently, because of neighboring relation of u, error is not propagated to the next steps of the recognition process 17 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Data Set Experiment • 30 gestures 10 repetitions each 18 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Result Experiment • SOM clustering – Blue: close to input vector – Red: not close • Recognition accuracy – Test with training data: 100% – 10-fold cross validation: 93% 0.843 ms for decoding a gesture – Only HMM-based classifier: 86.36% 19 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Conclusion • Key features – SOM and HMM based automatic recognition architecture – ROI • Relative hand position • Moving direction • Similarity of pattern • Application – Sign language – Gaming environment 20 S FT COMPUTING @ YONSEI UNIV . KOREA 16 Thank you