Video Expression Recognition Method Based on Spatiotemporal Recurrent Neural Network and Feature Fusion = Video Expression Recognition Method Based on Spatiotemporal Recurrent Neural Network and Feature Fusion

다국어 초록 (Multilingual Abstract)

Automatically recognizing facial expressions in video sequences is a challenging task because there is little direct correlation between facial features and subjective emotions in video. To overcome the problem, a video facial expression recognition method using spatiotemporal recurrent neural network and feature fusion is proposed. Firstly, the video is preprocessed. Then, the double-layer cascade structure is used to detect a face in a video image. In addition, two deep convolutional neural networks are used to extract the time-domain and airspace facial features in the video. The spatial convolutional neural network is used to extract the spatial information features from each frame of the static expression images in the video. The temporal convolutional neural network is used to extract the dynamic information features from the optical flow information from multiple frames of expression images in the video. A multiplication fusion is performed with the spatiotemporal features learned by the two deep convolutional neural networks. Finally, the fused features are input to the support vector machine to realize the facial expression classification task. The experimental results on cNTERFACE, RML, and AFEW6.0 datasets show that the recognition rates obtained by the proposed method are as high as 88.67%, 70.32%, and 63.84%, respectively. Comparative experiments show that the proposed method obtains higher recognition accuracy than other recently reported methods.

참고문헌 (Reference)

1 Y. Zhou, "The LAP under facility disruptions during early post-earthquake rescue using PSOGA hybrid algorithm" 28 (28): 9906-9914, 2019

2 S. Yu, "Single image dehazing using multiple transmission layer fusion" 63 (63): 519-535, 2016

3 Z. Gong, "Sequential data classification by dynamic state warping" 57 (57): 545-570, 2018

4 Y. Chen, "Robust and energy-efficient expression recognition based on improved deep ResNets" 64 (64): 519-528, 2019

5 L. H. Nguyen, "Real-time anomaly detection with Bayesian dynamic linear models" 26 (26): 2019

6 M. S. Hossain, "Real time facial expression recognition for nonverbal communication" 15 (15): 278-288, 2018

7 N. P. Gopalan, "Pattern averaging technique for facial expression recognition using support vector machines" 9 : 27-33, 2018

8 N. Wang, "Parsimonious extreme learning machine using recursive orthogonal least squares" 25 (25): 1828-1841, 2014

9 H. Li, "Multimodal 2D+ 3D facial expression recognition with deep fusion convolutional neural network" 19 (19): 2816-2831, 2017

10 H. Boughrara, "MLP neural network using modified constructive training algorithm : application to face recognition" 16 (16): 53-79, 2017

11 J. Zhao, "Learning deep facial expression features from image and optical flow sequences using 3D CNN" 34 (34): 1461-1475, 2018

12 N. Jain, "Hybrid deep neural networks for face emotion recognition" 115 : 101-106, 2018

13 M. U. Nagaral, "Hybrid approach for facial expression recognition using HJDLBP and LBP histogram in video sequences" 10 (10): 1-9, 2018

14 F. Khan, "Facial expression recognition using facial landmark detection and feature extraction via neural networks"

15 A. Moeini, "Facial expression recognition using dual dictionary learning" 45 : 20-33, 2017

16 F. Ahmed, "Facial expression recognition under difficult conditions : a comprehensive study on edge directional texture patterns" 28 (28): 399-409, 2018

17 E. Zangeneh, "Facial expression recognition by using differential geometric features" 66 (66): 463-470, 2018

18 E. Owusu, "Face detection based on multilayer feed-forward neural network and Haar features" 49 (49): 120-129, 2019

19 Y. Huang, "Expression-targeted feature learning for effective facial expression recognition" 55 : 677-687, 2018

20 S. Yuan, "Exponential elastic preserving projections for facial expression recognition" 275 : 711-724, 2018

21 M. Li, "Epidemic forest : a spatiotemporal model for communicable diseases" 109 (109): 812-836, 2019

22 O. Yi, "Dynamic games with asymmetric information : common information based perfect Bayesian equilibria and sequential decomposition" 62 (62): 222-237, 2016

23 Z. Yu, "Deeper cascaded peak-piloted network for weak expression recognition" 34 (34): 1691-1699, 2018

24 Z. Sun, "Combining the kernel collaboration representation and deep subspace learning for facial expression recognition" 27 (27): 2018

25 H. Yan, "Collaborative discriminative multi-metric learning for facial expression recognition in video" 75 : 33-40, 2018

26 J. Li, "CNN-based facial expression recognition from annotated rgb-d images for human–robot interaction" 16 (16): 2019

27 A. M. Shabat, "Angled local directional pattern for texture analysis with an application to facial expression recognition" 12 (12): 603-608, 2018

28 X. Liu, "Adaptive metric learning with deep neural networks for video-based facial expression recognition" 27 (27): 2008

29 J. Jian, "A multi-objective optimization model for green supply chain considering environmental benefits" 11 (11): 2019

30 X. Fan, "A discriminative dynamic framework for facial expression recognition in video sequences" 56 : 182-187, 2018