Self-feeding Semi-supervised Training Method for Grammatical Error Correction

다국어 초록 (Multilingual Abstract)

Grammatical error correction (GEC) has been successful with deep and complex neural machine translation models, but published annotated datasets to train the large models are scarce. In this dissertation, I propose a novel self-feeding training method that generates incorrect sentences from correct sentences. The proposed training method can generate appropriate wrong sentences from unlabeled sentences, using a data generation model trained as an autoencoder. It can also add artificial noise to correct sentences to automatically generate noisy sentences. I show that the GEC models trained with the self-feeding training method are successful without extra annotated data or deeper neural network-based models, achieving F0.5 score of 0.5982 on the CoNLL-2014 Shared Task test data with a transformer model. The results also show that fully unlabeled training is possible for data-scarce domains and languages.

목차 (Table of Contents)

• I. Introduction 1
• II. Related Studies 4
• III. Self-Feeding Training 7
• 3.1 Model-based data generation . . . . . . . . . . . . . . . . . . . . . . 8
• 3.2 Candidate selection process . . . . . . . . . . . . . . . . . . . . . . . 10
• 3.3 Unlabeled denoising training . . . . . . . . . . . . . . . . . . . . . . 12
• 3.4 Self-feeding training . . . . . . . . . . . . . . . . . . . . . . . . . . 15
• IV. Semi-Supervised Training of Self-Feeding Model 16
• 4.1 Mixed training . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
• 4.2 Interleaved training . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
• 4.3 Fine-tuned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
• V. Experiments 18
• 5.1 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
• 5.1.1 NUCLE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
• 5.1.2 CLC FCE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
• 5.1.3 W&I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
• 5.1.4 Lang-8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
• 5.1.5 Data preprocessing . . . . . . . . . . . . . . . . . . . . . . . 22
• 5.1.6 Unlabeled data . . . . . . . . . . . . . . . . . . . . . . . . . 23
• 5.1.7 Evaluation datasets . . . . . . . . . . . . . . . . . . . . . . . 24
• 5.2 Grammatical error correction models . . . . . . . . . . . . . . . . . . 24
• 5.3 Data Generation Methods . . . . . . . . . . . . . . . . . . . . . . . . 25
• 5.4 Model-based data generation . . . . . . . . . . . . . . . . . . . . . . 26
• 5.5 Self-feeding training . . . . . . . . . . . . . . . . . . . . . . . . . . 30
• 5.6 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
• 5.7 Results on the self-feeding method . . . . . . . . . . . . . . . . . . . 31
• 5.8 More experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
• 5.9 Fully unsupervised training . . . . . . . . . . . . . . . . . . . . . . . 34
• VI. Conclusions 35
• VII. Appendices 37
• 7.1 Data Generation Examples . . . . . . . . . . . . . . . . . . . . . . . 37
• 7.2 CoNLL result comparison . . . . . . . . . . . . . . . . . . . . . . . 41
• Summary (in Korean) 43
• References 44