Visual Speech Recognition of Korean Words Using Convolutional Neural Network

다국어 초록 (Multilingual Abstract)

In recent studies, speech recognition performance is greatly improved by using HMM and CNN. HMM is studying statistical modeling of voice to construct an acoustic model and to reduce the error rate by predicting voice through image of mouth region using CNN. In this paper, we propose visual speech recognition (VSR) using lip images. To implement VSR, we repeatedly recorded three subjects speaking 53 words chosen from an emergency medical service vocabulary book. To extract images of consonants, vowels, and final consonants in the recorded video, audio signals were used. The Viola–Jones algorithm was used for lip tracking on the extracted images. The lip tracking images were grouped and then classified using CNNs. To classify the components of a syllable including consonants, vowels, and final consonants, the structure of the CNN used VGG-s and modified LeNet-5, which has more layers. All syllable components were classified, and then the word was found by the Euclidean distance. From this experiment, a classification rate of 72.327% using 318 total testing words was obtained when VGG-s was used. When LeNet-5 applied this classifier for words, however, the classification rate was 22.327%.

참고문헌 (Reference)

1 유제훈, "컨볼루셔널 신경망과 케스케이드 안면 특징점 검출기를 이용한 얼굴의 특징점 분류" 제어·로봇·시스템학회 22 (22): 241-246, 2016

2 권용문, "양순음화의 발달" 한국어학회 47 : 93-130, 2010

3 S. S. Kumaravel, "Visual speech recognition using histogram of oriented displacements" Clemson University 2015

4 K. Simonyan, "Very deep convolutional networks for large-scale image recognition"

5 K. O. Yi, "The internal structure of Korean syllables: rhyme or body" 10 (10): 67-83, 1998

6 Jongju Shin, "Real-time lip reading system for isolated Korean word recognition" Elsevier BV 44 (44): 559-571, 2011

7 P. Viola, "Rapid object detection using a boosted cascade of simple features" 511-518, 2001

8 A. Vedaldi, "MatConvNet: convolutional neural networks for Matlab" a689-692, 2015

9 A. Krizhevsky, "ImageNet classification with deep convolutional neural networks" 25 : 1097-1105, 2012

10 Y. Lecun, "Gradient-based learning applied to document recognition" Institute of Electrical and Electronics Engineers (IEEE) 86 (86): 2278-2324, 1998

11 C. Szegedy, "Going deeper with convolutions" 1-9, 2015

12 J. Long, "Fully convolutional networks for semantic segmentation" 3431-3440, 2015

13 National Emergency Medical Center, "Emergency Medical Dictionary" National Emergency Medical Center 2005

14 C. Bregler, "Eigenlips” for robust speech recognition" 669-672, 1994

15 유제훈, "Cascade 안면 검출기와 컨볼루셔널 신경망을 이용한 얼굴 분류" 한국지능시스템학회 26 (26): 70-75, 2016

16 Kuniaki Noda, "Audio-visual speech recognition using deep learning" Springer Science and Business Media LLC 42 (42): 722-737, 2015

17 M. L. Seltzer, "An investigation of deep neural networks for noise robust speech recognition" 2013 : 7398-7402, 2013

18 Jinyu Li, "An Overview of Noise-Robust Automatic Speech Recognition" Institute of Electrical and Electronics Engineers (IEEE) 22 (22): 745-777, 2014

19 Ziheng Zhou, "A review of recent advances in visual speech decoding" Elsevier BV 32 (32): 590-605, 2014