Improving Abstractive Summarization by Training Masked Out-of-Vocabulary Words

다국어 초록 (Multilingual Abstract)

Text summarization is the task of producing a shorter version of a long document while accurately preservingthe main contents of the original text. Abstractive summarization generates novel words and phrases using alanguage generation method through text transformation and prior-embedded word information. However,newly coined words or out-of-vocabulary words decrease the performance of automatic summarization becausethey are not pre-trained in the machine learning process. In this study, we demonstrated an improvement insummarization quality through the contextualized embedding of BERT with out-of-vocabulary masking. Inaddition, explicitly providing precise pointing and an optional copy instruction along with BERT embedding,we achieved an increased accuracy than the baseline model. The recall-based word-generation metric ROUGE-1 score was 55.11 and the word-order-based ROUGE-L score was 39.65.

참고문헌 (Reference)

1 J. Howard, "Universal language model fine-tuning for text classification" 328-339, 2018

2 M. Allahyari, "Text summarization techniques: a brief survey"

3 G. Huang, "Snapshot ensembles: train 1, get m for free"

4 I. Sutskever, "Sequence to sequence learning with neural networks" 27 : 3104-3112, 2014

5 S. Xu, "Self-attention guided copy mechanism for abstractive summarization" 1355-1362, 2020

6 M. Yasunaga, "ScisummNet : a large annotated corpus and content-impact models for scientific paper summarization with citation networks" 7386-7393, 2019

7 I. Loshchilov, "SGDR: stochastic gradient descent with warm restarts"

8 K. Ganesan, "Rouge 2.0: updated and improved measures for evaluation of summarization tasks"

9 S. Narayan, "Ranking sentences for extractive summarization with reinforcement learning"

10 C. Gulcehre, "Pointing the unknown words" 140-149, 2016

11 J. Cheng, "Neural summarization by extracting sentences and words" 484-494, 2016

12 D. Bahdanau, "Neural machine translation by jointly learning to align and translate"

13 Tian Shi, "Neural Abstractive Text Summarization with Sequence-to-Sequence Models" Association for Computing Machinery (ACM) 2 (2): 1-37, 2021

14 J. Gu, "Incorporating copying mechanism in sequence-to-sequence learning" 1631-1640, 2016

15 L. N. Smith, "Cyclical learning rates for training neural networks" 464-472, 2017

16 W. Wang, "Concept pointer network for abstractive summarization" 3076-3085, 2019

17 G. Rossiello, "Centroid-based text summarization through compositionality of word embeddings" 12-21, 2017

18 J. Devlin, "BERT: pre-training of deep bidirectional transformers for language understanding"

19 이태석 ; 강승식, "BERT 임베딩과 선택적 OOV 복사 방법을 사용한 문서요약" 한국정보과학회 47 (47): 36-44, 2020

20 M. Hu, "Attention-guided answer distillation for machine reading comprehension"

21 A. Vaswani, "Attention is all you need" 30 : 5998-6008, 2017

22 R. Nallapati, "Abstractive text summarization using sequence-tosequence RNNs and beyond"

23 Y. Dong, "A survey on neural network-based summarization methods"

24 N. Nazari, "A survey on automatic text summarization" 7 (7): 121-135, 2019

25 S. Erera, "A summarization system for scientific documents" 211-216, 2019

26 K. Goyal, "A continuous relaxation of beam search for end-toend training of neural sequence models" 3045-3052, 2018