A Large-scale Text Analysis with Word Embeddings and Topic Modeling = A Large-scale Text Analysis with Word Embeddings and Topic Modeling

다국어 초록 (Multilingual Abstract)

This research exemplifies how statistical semantic models and word embedding techniques can play a role in understanding the system of human knowledge. Intuitively, we speculate that when a person is given a piece of text, they first classify the semantic contents, group them to semantically similar texts previously observed, then relate their contents with the group. We attempt to model this process of knowledge linking by using word embeddings and topic modeling. Specifically, we propose a model that analyzes the semantic/thematic structure of a given corpus, so as to replicate the cognitive process of knowledge ingestion. Our model attempts to make the best of both word embeddings and topic modeling by first clustering documents and then performing topic modeling on them. To demonstrate our approach, we apply our method to the Corpus of Contemporary American English (COCA). In COCA, the texts are first divided by text type and then by subcategory, which represents the specific topics of the documents. To show the effectiveness of our analysis, we specifically focus on the texts related to the domain of science. First, we cull out science-related texts from various genres, then preprocess the texts into a usable, appropriate format. In our preprocessing steps, we attempt to fine-grain the texts with a combination of tokenization, parsing, and lemmatization. Through this preprocess, we discard words of little semantic value and disambiguate syntactically ambiguous words. Afterwards, using only the nouns from the corpus, we train a word2vec model on the documents and apply K-means clustering to them. The results from clustering show that each cluster represents each branch of science, similar to how people relate a new piece of text to semantically related documents. With these results, we proceed on to perform topic modeling on each of these clusters, which reveal latent topics cluster and their relationship with each other. Through this research, we demonstrate a way to analyze a mass corpus and highlight the semantic/ thematic structure of topics in it, which can be thought as a representation of knowledge in human cognition.

참고문헌 (Reference)

1 홍정하, "토픽모델링을 이용한 코퍼스의 주제구조 탐색" 언어정보연구소 30 : 239-276, 2017

2 Manning, Christopher D., "The Stanford CoreNLP Natural Language Processing Toolkit" 55-60, 2014

3 Radim Rehurek, "Software Framework for Topic Modelling with Large Corpora" 2010

4 Pedregosa, Fabian, "Scikit-learn: Machine Learning in Python" 2011

5 Bird, Steven, "Natural Language Processing with Python" O’Reilly Media Inc 2009

6 Blei, David M., "Latent Dirichlet Allocation" 3 : 993-1022, 2003

7 Sievert, Carson, "LDAvis: A method for visualizing and interpreting topics" 2014

8 Alex Wang, "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding" Google DeepMind & NYU 2018

9 AlexWang, Amapreet Singh, "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding"

10 Griffiths, Thomas L., "Finding Scientific Topics" PNAS 2004

11 Mikolov, Tomas, "Efficient Estimation of Word Representations in Vector Space" NIPS 2013

12 "COCA"