작물 분류에서 시공간 특징을 고려하기 위한 2D CNN과 양방향 LSTM의 결합 = Combining 2D CNN and Bidirectional LSTM to Consider Spatio-Temporal Features in Crop Classification

다국어 초록 (Multilingual Abstract)

In this paper, a hybrid deep learning model, called 2D convolution with bidirectional long short-term memory (2DCBLSTM), is presented that can effectively combine both spatial and temporal features for crop classification. In the proposed model, 2D convolution operators are first applied to extract spatial features of crops and the extracted spatial features are then used as inputs for a bidirectional LSTM model that can effectively process temporal features. To evaluate the classification performance of the proposed model, a case study of crop classification was carried out using multi-temporal unmanned aerial vehicle images acquired in Anbandegi, Korea. For comparison purposes, we applied conventional deep learning models including two-dimensional convolutional neural network (CNN) using spatial features, LSTM using temporal features, and three-dimensional CNN using spatio-temporal features. Through the impact analysis of hyper-parameters on the classification performance, the use of both spatial and temporal features greatly reduced misclassification patterns of crops and the proposed hybrid model showed the best classification accuracy, compared to the conventional deep learning models that considered either spatial features or temporal features. Therefore, it is expected that the proposed model can be effectively applied to crop classification owing to its ability to consider spatio-temporal features of crops.

국문 초록 (Abstract)

이 논문에서는 작물 분류를 목적으로 작물의 시공간 특징을 고려할 수 있는 딥러닝 모델 2D convolution with bidirectional long short-term memory(2DCBLSTM)을 제안하였다. 제안 모델은 우선 작물의 공간 특징을 추출하기 위해 2차원의 합성곱 연산자를 적용하고, 추출된 공간 특징을 시간 특징을 고려할 수 있는 양방향 LSTM 모델의 입력 자료로 이용한다. 제안 모델의 분류 성능을 평가하기 위해 안반덕에서 수집된 다중시기 무인기 영상을 이용한 밭작물 구분 사례 연구를 수행하였다. 비교를 목적으로 기존 딥러닝 모델인 2차원의 공간 특징을이용하는 2D convolutional neural network(CNN), 시간 특징을 이용하는 LSTM과 3차원의 시공간 특징을 이용하는 3D CNN을 적용하였다. 하이퍼 파라미터의 영향 분석을 통해, 시공간 특징을 이용함으로써 작물의 오분류 양상을 현저히 줄일 수 있었으며, 제안 모델이 공간 특징이나 시간 특징만을 고려하는 기존 딥러닝 모델에비해 가장 우수한 분류 정확도를 나타냈다. 따라서 이 연구에서 제안된 모델은 작물의 시공간 특징을 고려할수 있기 때문에 작물 분류에 효과적으로 적용될 수 있을 것으로 기대된다.

참고문헌 (Reference)

1 김예슬, "작물분류에서 기계학습 및 딥러닝 알고리즘의 분류 성능 평가: 하이퍼파라미터와 훈련자료 크기의 영향 분석" 대한원격탐사학회 34 (34): 811-827, 2018

2 송아람, "딥러닝 기반의 초분광영상 분류를 사용한 환경공간정보시스템 활용" 대한원격탐사학회 33 (33): 1061-1073, 2017

3 이성혁, "딥러닝 기반의 영상분할을 이용한 토지피복분류" 대한원격탐사학회 35 (35): 279-288, 2019

4 Russwurm, M., "Temporal vegetation modelling using long short-term memory networks for crop identification from medium-resolution multi-spectral satellite images" 11-19, 2017

5 Kim, Y., "Selflearning based land-cover classification using sequential class patterns from past land-cover maps" 9 (9): 921-, 2017

6 Hua, Y., "Recurrently exploring class-wise attention in a hybrid convolu-tional and bidirectional LSTM network for multilabel aerial image classification" 149 : 188-199, 2019

7 Kussul, N., "Parcelbased crop classification in Ukraine using Landsat-8 data and Sentinel-1A data" 9 (9): 2500-2508, 2016

8 Chiu, J. P. C., "Named entity recognition with bidirectional LSTM-CNNs" 4 : 357-370, 2016

9 Feng, Q., "Multisource hyperspectral and LiDAR data fusion for urban land-use mapping based on a modified two-branch convolutional neural network" 8 (8): 28-, 2019

10 Russwurm, M., "Multi-temporal land cover classification with sequential recurrent encoders" 7 (7): 129-, 2018

11 Wei, S., "Multi-temporal SAR data large-scale crop mapping based on U-Net model" 11 (11): 68-, 2019

12 Mou, L., "Learning spectral-spatial-temporal features via a recurrent convolutional neural network for change detection in multispectral imagery" 57 (57): 924-935, 2018

13 Kwak, G. -H., "Impact of texture information on crop classification with machine learning and UAV images" 9 (9): 643-, 2019

14 Zhou, F., "Hyperspectral image classification using spectralspatial LSTMs" 328 : 39-47, 2019

15 Ma, A., "Hyperspectral image classification using similarity measurements-based deep recurrent neural networks" 11 (11): 194-, 2019

16 Zhong, L., "Deep learning based multi-temporal crop classification" 221 : 430-443, 2019

17 Xie, B., "Deep convolutional neural network for mapping smallholder agriculture using high spatial resolution satellite image" 19 (19): 2398-, 2019

18 Tatsumi, K., "Crop classification of upland fields using Random forest of time-series Landsat 7 ETM+ data" 115 : 171-179, 2015

19 Liu, Q., "Bidirectional-convolutional LSTM based spectralspatial feature learning for hyperspectral image classification" 9 (9): 1330-, 2017

20 Schuster, M., "Bidirectional recurrent neural networks" 45 (45): 2673-2681, 1997

21 Ullah, A., "Action recognition in video sequences using deep bi-directional LSTM with CNN features" 6 : 1155-1166, 2017

22 Zhang, C., "A hybrid MLPCNN classifier for very fine resolution remotely sensed image classification" 140 : 133-144, 2018

23 Siachalou, S., "A hidden Markov models approach for crop classification : Linking crop phenology to time series of multi-sensor remote sensing data" 7 (7): 3633-3650, 2015

24 Ji, S., "3D convolutional neural networks for crop classification with multi-temporal remote sensing images" 10 (10): 75-, 2018

25 Seydgar, M., "3-D convolution-recurrent networks for spectral-spatial classification of hyperspectral images" 11 (11): 883-, 2019