Emotion and intention recognition using facial images and its application to ADAS

국문 초록 (Abstract)

본 논문에서는 얼굴 영상에서 생체신호 및 다양한 정보를 추출하여 감정 인식을 수행하고 인식을 하고 행동 영상을 촬영하여 행동인식을 한 뒤 이 둘을 결합하여 최종적으로 사용자의 의도를 인식하는 방법에 대해 연구한다. 기존에 사용자의 의도를 분석하는 연구는 긍정, 부정 감정에 대한 스코어를 제시하지 못했으며 대부분 하나 또는 두 감정을 사용하여 긍정, 부정을 분류하였고 설문지를 통해서만 그들의 감정을 분석하였다. 따라서 본 연구는 의도에 가장 영향을 끼치는 요소들을 분석하여 감정과 행동 두 가지를 선정하였고 이를 의도 인식에 활용하였다. 기존 연구에서 졸음이나 시선인식으로 운전자의 의도를 파악한 것과 다르게 감정과 행동을 인식 및 결합하여 운전자의 의도를 파악할 수 있게 한다. 7감정을 사용하여 긍정, 부정, 중립의 감정을 인식하고 행동 인식과 더불어 사용자의 의도를 인식했으며 이를 자율 주행 자동차에 적용하여 의도 인식 시스템을 제안 및 개발하였다.
본 논문에서는 얼굴 영상에서 7감정을 인식하고, 행동 영상에서 행동을 인식하 여 의도 인식 방법을 제안하였고 이를 자율주행 차량 환경에 적용하였다.

다국어 초록 (Multilingual Abstract)

In this paper, we study the method of recognizing user's intention by combining emotional recognition and behavior recognition. The previous studies that analyzed the intention of the users did not provide a score indicating the degree of positive and negative feelings and additional things. Mostly, one or two feelings were used to classify positive and negative emotion and just using questionnaires to recognized users intention. Therefore, this study analyzed the factors that have the greatest effect on intention and we selected two factors, emotion and behavior. In existed study, driver’s intention was recognized by drowsiness or gaze recognition however, in this study it can be carried out by combining emotion and behavior recognition.
In this paper, we implemented an intention recognition system by recognizing 7 emotions using face image and recognizing driver’s behaviors then applied it to autonomous vehicle environment.

목차 (Table of Contents)

• List of Tables············iii
• List of Figures ············v
• Abstract ············vi
• 1. Introduction············1
• 2. Related Works············3
• 2.1 Extracting bio-signal in face image············3
• 2.1.1 Pulse rate Estimation using Face Image············3
• 2.1.2 Skin Moisture and Texture Estimation using Face
• Image············5
• 2.2 Emotion recognition using bio-signal············6
• 2.2.1 Emotion recognition using PPG············6
• 2.2.2 Positive and Negative Emotion Recognition using
• bio-signal············8
• 2.3 Emotion recognition using Face Image············8
• 2.3.1 Emotion recognition using Face expression············8
• 2.3.2 Emotion recognition using Deep-Learning············10
• 2.4 Intention recognition············11
• 2.4.1 Intention recognition using Face Image············11
• 2.4.2 Intention recognition through positive and negative
• feelings············12
• 2.4.3 Intention recognition in driving status············13
• 3. Emotion and Intention recognition using Face Image············15
• 3.1 Extracting Bio-signal in Face Image············15
• 3.1.1 Pulse rate Extraction············15
• 3.1.2 Moisture Extraction············16
• 3.1.3 PRV and Rate of PRV change Extraction············17
• 3.2 Emotion Recognition using Bio-signal············18
• 3.2.1 Multidimensional Emotion Model············18
• 3.2.2 Generating individual and whole emotion models
• using extracted bio-signals············19
• 3.2.3 Emotion Recognition using bio-signal············20
• 3.2.4 Emotion Recognition using color and texture
• information············21
• 3.2.5 Emotion Recognition using combination of color,
• texture and bio-signal············22
• 3.2.6 Positive and Negative Emotion Recognition using
• bio-signal············23
• 3.3 Emotion Recognition using Deep-Learning············24
• 3.3.1 Generating test and validation data set with
• deep-learning············24
• 3.3.2 Emotion Recognition using deep-learning············24
• 3.3.3 Positive and Negative Emotion Recognition using
• deep-learning ············26
• 3.4 Intention recognition using Face Image············27
• 3.4.1 Basic survey for intention recognition············27
• 3.4.2 Driver status recognition············29
• 3.4.3 Intention Recognition using combination of driver
• status and positive and negative emotion············31
• 3.4.4 Application of Intention Recognition in driving status
• ············34
• 4. Experimental and Results············40
• 4.1 Extracting Bio-signal using Face Image············40
• 4.1.1 Experimental Environment············40
• 4.1.2 Experiment Results············40
• 4.2 Emotion Recognition using Face Image············41
• 4.2.1 Experimental Environment············41
• 4.2.2 Experiment Results············42
• 4.3 Intention Recognition using Face Image ············47
• 4.3.1 Experimental Environment············47
• 4.3.2 Experiment Results············49
• 5. Conclusion············51
• References············54
• Korean Abstract············57