Sentence-Chain Based Seq2seq Model for Corpus Expansion

다국어 초록 (Multilingual Abstract)

This study focuses on a method for sequential data augmentation in order to alleviate data sparseness problems. Specifically, we present corpus expansion techniques for enhancing the coverage of a language model. Recent recurrent neural network studies show that a seq2seq model can be applied for addressing language generation issues; it has the ability to generate new sentences from given input sentences. We present a method of corpus expansion using a sentence-chain based seq2seq model. For training the seq2seq model, sentence chains are used as triples. The first two sentences in a triple are used for the encoder of the seq2seq model, while the last sentence becomes a target sequence for the decoder. Using only internal resources, evaluation results show an improvement of approximately 7.6% relative perplexity over a baseline language model of Korean text. Additionally, from a comparison with a previous study, the sentence chain approach reduces the size of the training data by 38.4% while generating 1.4-times the number of n-grams with superior performance for English text.

참고문헌 (Reference)

1 R. Bhagat, "What is a Paraphrase?" 39 (39): 463-472, 2013

2 R. Barzilay, "Using Lexical Chains for Text Summarization" 10-17, 1997

3 B. Dolan, "Unsupervised Construction of Large Paraphrase Corpora: Exploiting Massively Parallel News Sources" 350-356, 2004

4 M. Abadi, "Tensorflow: Large-scale machine learning on heterogeneous distributed systems"

5 K.M. Hermann, "Teaching Machines to Read and Comprehend" 1693-1701, 2015

6 A.F. Smeaton, "TREC-4Experiments at Dublin City University: Thresholding Posting Lists, Query Expansion With WordNet and POS Tagging of Spanish" 373-389, 1995

7 R. Kiros, "Skip-Thought Vectors" 3294-3302, 2015

8 I. Sutskever, "Sequence to sequence learning with neural networks" 3104-3112, 2014

9 A.M. Dai, "Semi-Supervised Sequence Learning" 3079-3087, 2015

10 F.H. Khan, "SWIMS: Semi-Supervised Subjective Feature Weighting and Intelligent Model Selection for Sentiment Analysis" 100 : 97-111, 2016

11 A. Stolcke, "SRILM – an Extensible Language Modeling Toolkit" 901-904, 2002

12 F. Jelinek, "Perplexity—A Measure of Difficulty of Speech Recognition Tasks" 1977

13 J. Bradbury, "MetaMind Neural Machine Translation System for WMT 2016" 264-267, 2016

14 S. Hochreiter, "Long Short-Term Memory" 9 (9): 1735-1780, 1997

15 M. Marathe, "Lexical Chains Using Distributional Measures of Concept Distance" 291-302, 2010

16 K. Cho, "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation" 1724-1734, 2014

17 K. Greff, "LSTM: A Search Space Odyssey"

18 D. Rey, "International Encyclopedia of Statistical Science" Springer Berlin Heidelberg 1658-1659, 2011

19 J.R. Smith, "Integrated Spatial and Feature Image System:Retrieval, Analysis and Compression" Columbia Univ. 1997

20 R. Sennrich, "Improving Neural Machine Translation Models with Monolingual Data"

21 A. Krizhevsky, "Imagenet Classification with Deep Convolutional Neural Networks" 1097-1105, 2012

22 B.J. Hsu, "Generalized linear interpolation of language models" 136-140, 2007

23 D. Zhang, "Evaluation of similarity measurement for image retrieval" 928-931, 2003

24 J. Zhao, "ECNU: Using Traditional Similarity Measurements and Word Embedding for Semantic Textual Similarity Estimation" 117-122, 2015

25 R. Socher, "Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection" 801-809, 2011

26 S. Remus, "Domain-Specific Corpus Expansion with Focused Webcrawling" 2016

27 E. Chung, "Domain-Adapted Word Segmentation for an Out-of-Domain Language Modeling" 63-73, 2011

28 T. Mikolov, "Distributed Representations of Words and Phrases and their Compositionality" 3111-3119, 2013

29 X. Cui, "Data Augmentation for Deep Neural Network Acoustic Modeling" 23 (23): 1469-1477, 2015

30 Q. Gao, "Corpus Expansion for Statistical Machine Translation with Semantic Role Label Substitution Rules" 294-298, 2011

31 M. Negri, "Chinese Whispers: Cooperative Paraphrase Acquisition" 2659-2665, 2012

32 X. Zhang, "Character-Level Convolutional Networks for Text Classification" 649-657, 2015

33 Y. Ma, "Bilingually Motivated Domain-Adapted Word Segmentation for Statistical Machine Translation" 549-557, 2009

34 B. Harb, "Back-off Language Model Compression" 353-355, 2009

35 X. Qiu, "Automatic Corpus Expansion for Chinese Word Segmentation by Exploiting the Redundancy of Web Information" 1154-1164, 2014

36 V.L. Colson, "Automated Call Center Transcription Services"

37 S. Zhao, "Application-Driven Statistical Paraphrase Generation" 834-842, 2009

38 A. Sordoni, "A Neural Network Approach to Context-Sensitive Generation of Conversational Responses" 196-205, 2015

39 O. Vinyals, "A Neural Conversational Model" 2015

40 J. Li, "A Diversity-Promoting Objective Function for Neural Conversation Models" 110-119, 2016