BERT 기반 Variational Inference와 RNN을 이용한 한국어 영화평 감성 분석 = Sentimental Analysis of Korean Movie Review using Variational Inference and RNN based on BERT

국문 초록 (Abstract)

최근 자연어처리 분야에서 많은 성능 향상을 보이고 있는 BERT는 양방향성을 가진 트랜스포머(transformer)를 기반으로 한 모델이다. BERT는 OOV (Out Of Vocabulary) 문제를 해결하기 위하여 BPE (Byte Pair Encoding)를 적용하며, 이를 기반으로 언어 모델을 사전 학습하고 출력 층(layer)을 추가하여 자연어처리 태스크를 fine-tuning한다. 감성 분석은 주어진 문장에 대한 잠재적 의미를 분석하고 분류하는 문제이다. 본 논문에서는 감성 분석에 BERT로부터 생성되는 토큰 표현을 이용하기 위하여 대용량 한국어 코퍼스로 언어 모델을 학습한 BERT 모델을 사용한다. 또한 문맥 정보를 인코딩하는 RNN을 BERT 함께 사용하는 방법과, RNN으로 인코딩한 hidden state에 variational inference를 이용하여 감성 분석을 수행하는 방법을 제안한다.

다국어 초록 (Multilingual Abstract)

BERT is a model based on a bidirectional transformer and has demonstrated immense improvement in performance in the field of natural language processing. BERT applies BPE (Byte Pair Encoding) to solve OOV (Out Of Vocabulary) problem, and pre-trains language model based on the BPE and fine-tunes natural language processing task by adding an output layer. Sentimental analysis is a task of analyzing and classifying the potential meaning of a given sentence. In the present work, we have employed the BERT model which trains the language model with large capacity Korean Corpus to use the token representation generated from BERT for sentimental analysis. In addition, we propose a method of using BERT along with RNN, which encodes context information and a method of performing sentimental analysis using variational inference in hidden state encoded by RNNv.

참고문헌 (Reference)

1 "https://github.com/huggingface/pytorch-pretrained-BERT"

2 "https://github.com/e9t/nsmc"

3 "http://aiopen.etri.re.kr/"

4 Jang Hwan-seok, "The method of analyzing sentiment keywords using Word2Vec" 661-663, 2017

5 Danilo Jimenez Rezende, "Stochastic backpropagation and approximate inference in deep generative models" 1278-1286, 2014

6 Tao Lei, "Simple Recurrent Units for Highly Parallelizable Recurrence" 4470-4481, 2018

7 Jun Young Youn, "Sentiment Analysis for Korean Product Review Using Stacked Bi-LSTM-CRF Model" 633 (633): 633-635, 2018

8 Yishu Miao, "Neural Variational inference for Text Processing"

9 Min Kim, "Multi-channel CNN for Korean Sentiment Analysis" 79-83, 2018

10 Sepp Hochreiter, "Long short-term memory" 9 (9): 1735-1780, 1997

11 Kyunghyun Cho, "Learning phrase representation using RNN encoder-decoder for statistical machine translation" 14 : 2014

12 H. Jang, "Language-Specific Sentiment Analysis in Morphologically Rich Languages" 498-506, 2010

13 Geon-Yeong Kim, "Korean Text Generation and Sentiment Analysis Using Model Combined VAE and CNN" 430-433, 2018

14 Yeongtaek Oh, "Korean Movie-review Sentiment Analysis Using Parallel Stacked Bidirectional LSTM Model" 823-825, 2018

15 Cheoneum Park, "Korean Movie Review Sentimental Analysis using RNN-based Variational Inference" 2018

16 Geonyeong Kim, "Korean Movie Review Sentiment Analysis Using Convolutional Neural Network" 747-749, 2016

17 Cheoneum Park, "Korean Dependency Parsing with BERT" 2019

18 M. Bautin, "International Sentiment Analysis for News and Blogs"

19 Alec Radford, "Improving Language Understanding by Generative Pre-Training"

20 D. Hendrycks, "Gaussian Error Linear Units (GELUs)"

21 X. Glorot, "Domain adaptation for large-scale sentiment classification : a deep learning approach" 513-520, 2011

22 Matthew E. Peters, "Deep contextualized word representations"

23 DaumKakao, "DaumKakao. movie review page"

24 Y. Kim, "Convolutional Neural Networks for Sentence Classification" 1746-1751, 2014

25 Cheoneum Park, "Contextualized Embedding-based Korean Movie Review Sentiment Analysis" 75-78, 2018

26 J. Devlin, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding"

27 Diederik P. Kingma, "Auto-Encoding Variational Bayes"

28 A. Vaswani, "Attention Is All You Need" 5998-6008, 2017