ESRGAN과 Semantic Soft Segmentation을 이용한 객체 분할

윤동식,곽노윤 한국사물인터넷학회 2023 한국사물인터넷학회 논문지 Vol.9 No.1

This paper is related to object segmentation using ESRGAN(Enhanced Super Resolution GAN) and SSS(Semantic Soft Segmentation). The segmentation performance of the object segmentation method using Mask R-CNN and SSS proposed by the research team in this paper is generally good, but the segmentation performance is poor when the size of the objects is relatively small. This paper is to solve these problems. The proposed method aims to improve segmentation performance of small objects by performing super-resolution through ESRGAN and then performing SSS when the size of an object detected through Mask R-CNN is below a certain threshold. According to the proposed method, it was confirmed that the segmentation characteristics of small-sized objects can be improved more effectively than the previous method. 본 논문은 ESRGAN(Enhanced Super Resolution GAN)과 SSS(Semantic Soft Segmentation)을 이용한객체 분할에 관한 것이다. 본 논문의 연구진이 앞서 제안한 Mask R-CNN과 SSS를 이용한 객체 분할 방법의 분할 성능은 전반적으로 양호하지만 객체의 크기가 상대적으로 작은 경우 분할 성능이 저조해지는 문제점이 있었다. 본 논문은이러한 문제점을 해소하기 위한 것이다. 제안된 방법은 Mask R-CNN을 통해 검출된 객체의 크기가 일정 기준치 이하인 경우, ESRGAN을 통해 초해상화를 수행한 후, SSS을 수행함으로써 소형 객체의 분할 성능을 개선하고자 한다. 제안된 방법에 따르면, 기존의 방법에 비해 크기가 작은 객체의 분할 특성을 좀 더 효과적으로 개선할 수 있음을 확인할수 있었다.

안개영상의 의미론적 분할 및 안개제거를 위한 심층 멀티태스크 네트워크

송태용,장현성,하남구,연윤모,권구용,손광훈 한국멀티미디어학회 2019 멀티미디어학회논문지 Vol.22 No.9

Image semantic segmentation and dehazing are key tasks in the computer vision. In recent years, researches in both tasks have achieved substantial improvements in performance with the development of Convolutional Neural Network (CNN). However, most of the previous works for semantic segmentation assume the images are captured in clear weather and show degraded performance under hazy images with low contrast and faded color. Meanwhile, dehazing aims to recover clear image given observed hazy image, which is an ill-posed problem and can be alleviated with additional information about the image. In this work, we propose a deep multi-task network for simultaneous semantic segmentation and dehazing. The proposed network takes single haze image as input and predicts dense semantic segmentation map and clear image. The visual information getting refined during the dehazing process can help the recognition task of semantic segmentation. On the other hand, semantic features obtained during the semantic segmentation process can provide cues for color priors for objects, which can help dehazing process. Experimental results demonstrate the effectiveness of the proposed multi-task approach, showing improved performance compared to the separate networks.

Multimodal sensor-based semantic 3D mapping for a large-scale environment

Jeong, Jongmin,Yoon, Tae Sung,Park, Jin Bae Elsevier 2018 expert systems with applications Vol.105 No.-

<P><B>Abstract</B></P> <P>Semantic 3D mapping is one of the most important fields in robotics, and has been used in many applications, such as robot navigation, surveillance, and virtual reality. In general, semantic 3D mapping is mainly composed of 3D reconstruction and semantic segmentation. As these technologies evolve, there has been great progress in semantic 3D mapping in recent years. Furthermore, the number of robotic applications requiring semantic information in 3D mapping to perform high-level tasks has increased, and many studies on semantic 3D mapping have been published. Existing methods use a camera for both 3D reconstruction and semantic segmentation. However, this is not suitable for large-scale environments and has the disadvantage of high computational complexity. To address this problem, we propose a multimodal sensor-based semantic 3D mapping system using a 3D Lidar combined with a camera. In this study, the odometry is obtained by high-precision global positioning system (GPS) and inertial measurement unit (IMU), and it is estimated by iterative closest point (ICP) when a GPS signal is weak. Then, we use the latest 2D convolutional neural network (CNN) for semantic segmentation. To build a semantic 3D map, we integrate the 3D map with semantic information by using coordinate transformation and Bayes’ update scheme. In order to improve the semantic 3D map, we propose a 3D refinement process to correct wrongly segmented voxels and remove traces of moving vehicles in the 3D map. Through experiments on challenging sequences, we demonstrate that our method outperforms state-of-the-art methods in terms of accuracy and intersection over union (IoU). Thus, our method can be used for various applications that require semantic information in 3D map.</P> <P><B>Highlights</B></P> <P> <UL> <LI> A novel method to generate semantic 3D map by combining a 3D Lidar and a camera for large-scale environments. </LI> <LI> A refinement method to remove traces of moving vehicles in a 3D map. </LI> <LI> Experiments on challenging sequences and real-world data to compare against state-of-the-art methods. </LI> <LI> Demonstration of superiority in terms of 3D accuracy and intersection over union (IoU). </LI> </UL> </P>

Development of Semantic Segmentation Algorithm for Crops with Long Narrow Leaves using UAV RGB Imagery

( Dong-wook Kim ),( Gyujin Jang ),( Hak-jin Kim ) 한국농업기계학회 2023 한국농업기계학회 학술발표논문집 Vol.28 No.1

Image segmentation is fundamental and important in agricultural crops using remote sensing technology. Many issues, such as crop growth stage prediction, crop line detection, crop density estimation, cover crop identification, leaf disease detection, and crop biomass monitoring, are highly dependent on the performance of crop segmentation algorithms. Under field conditions, crop segmentation for unmanned aerial vehicle (UAV) imagery should be more sophisticated considering geometric distortion of images by wind and illumination variations. In Korean crops, a plastic mulch used to restrict weeds and prevent cold weather damage makes the background more complex. In addition, in the presence of various Korean crops grown in the field, the boundary between crops and background may become unclear and complicated due to varying sunlight conditions that generate shadows and reflections. In our previous study, excess green and CIE L*a*b* color space were used for the segmentation of four different crops, i.e., Chinese cabbage, white radish, onion, and garlic. However, it was reported that the segmentation performance was strongly limited when crops with long and narrow leaves, such as onion and garlic, were in the early growth stages owing to higher effect of shadow. In this study, a CNN-based crop segmentation algorithm was developed to effectively separate crops from background consisting of soil, plastic mulch from UAV RGB images. A two-year field experiment was conducted during the 2017 and 2018 growing seasons to validate learning model using a separate dataset. RGB images were collected using the UAV flying over the test fields at 10 m above ground level (AGL) on almost 1-week interval. The RGB-based a* band images were exquisitely binarized into crops and backgrounds using manual threshold. The binary images were used as an annotation file showing the positions of crops and backgrounds to construct training data for semantic segmentation. The orthomosaic UAV images from early to late growth stages were cropped to a size of 256 by 256 pixels. The U-Net model was developed using 70% of the 2017 data for training and 30% for validation. The developed model was tested using the 2018 data. As a result, the average segmentation accuracy was 90.1%. In further studies, it is expected that crop segmentation performance can be improved by increasing the accuracy of the a*band-based annotation file.

DA-Res2Net: a novel Densely connected residual Attention network for image semantic segmentation

( Xiaopin Zhao ),( Weibin Liu ),( Weiwei Xing ),( Xiang Wei ) 한국인터넷정보학회 2020 KSII Transactions on Internet and Information Syst Vol.14 No.11

Since scene segmentation is becoming a hot topic in the field of autonomous driving and medical image analysis, researchers are actively trying new methods to improve segmentation accuracy. At present, the main issues in image semantic segmentation are intra-class inconsistency and inter-class indistinction. From our analysis, the lack of global information as well as macroscopic discrimination on the object are the two main reasons. In this paper, we propose a Densely connected residual Attention network (DA-Res2Net) which consists of a dense residual network and channel attention guidance module to deal with these problems and improve the accuracy of image segmentation. Specifically, in order to make the extracted features equipped with stronger multi-scale characteristics, a densely connected residual network is proposed as a feature extractor. Furthermore, to improve the representativeness of each channel feature, we design a Channel-Attention-Guide module to make the model focusing on the high-level semantic features and low-level location features simultaneously. Experimental results show that the method achieves significant performance on various datasets. Compared to other state-of-the-art methods, the proposed method reaches the mean IOU accuracy of 83.2% on PASCAL VOC 2012 and 79.7% on Cityscapes dataset, respectively.

Skin Lesion Segmentation with Codec Structure Based Upper and Lower Layer Feature Fusion Mechanism

( Cheng Yang ),( Guanming Lu ) 한국인터넷정보학회 2022 KSII Transactions on Internet and Information Syst Vol.16 No.1

The U-Net architecture-based segmentation models attained remarkable performance in numerous medical image segmentation missions like skin lesion segmentation. Nevertheless, the resolution gradually decreases and the loss of spatial information increases with deeper network. The fusion of adjacent layers is not enough to make up for the lost spatial information, thus resulting in errors of segmentation boundary so as to decline the accuracy of segmentation. To tackle the issue, we propose a new deep learning-based segmentation model. In the decoding stage, the feature channels of each decoding unit are concatenated with all the feature channels of the upper coding unit. Which is done in order to ensure the segmentation effect by integrating spatial and semantic information, and promotes the robustness and generalization of our model by combining the atrous spatial pyramid pooling (ASPP) module and channel attention module (CAM). Extensive experiments on ISIC2016 and ISIC2017 common datasets proved that our model implements well and outperforms compared segmentation models for skin lesion segmentation.

PointMS: Semantic Segmentation for Point Cloud Based on Multi-scale Directional Convolution

Hui Chen,Wanlou Chen,Yipeng Zuo,Peng Xu,Zhonghua Hao 제어·로봇·시스템학회 2022 International Journal of Control, Automation, and Vol.20 No.10

In the field of point cloud scene segmentation with deep learning, the ability of the network to extract spatial structure information limits the performance of semantic segmentation. This work proposes a novel framework named PointMS, which handles the semantic segmentation of point cloud scene, to solve the problem of missing local feature information due to the lack of spatial structure information on the training stage. The structure of framework utilizes spatial structure information of point cloud and balances the extraction of global feature and subtle feature when processing point cloud data. Firstly, a multi-scale combination module (SIFT-MS) is used to extract local features of different scales for enhancing the perception of local structure information at each point. Secondly, the process of feature transmission often leads to the loss of information, so a feature supplement module (FSM) is proposed to complete the information lost after feature transformation through the effective combination of global feature and subtle feature. This module integrates the features of different locations to supplement the information lost in feature conversion. The experimental results demonstrate that the proposed framework is efficient for semantic segmentation of S3DIS dataset. SIFT-MS module and FSM module can effectively improve the performance of the semantic segmentation model of point cloud.

영상처리 및 심층 네트워크 기반 영상 분할 방법

정홍구,정현우,윤병현,최강선 대한전자공학회 2021 전자공학회논문지 Vol.58 No.1

Deep neural network (DNN)-based image segmentation has performed well at the image semantic segmentation challenges. However, iterative sampling in the network causes inaccurate and uncertain segment boundaries. On the other hand, conventional image segmentation based on image processing tends to extract well-aligned object boundaries, but has been difficult in the semantic segmentation. In this paper, we propose image segmentation method that exploits both image processing techniques and the DNN to extract semantic objects with well-aligned boundaries. In the experiments, it is confirmed that the proposed algorithm improves DNN-based segmentation effectively. 심층 신경망 이용하는 영상 분할 방법은 의미 단위 영상 분할 문제에서 좋은 성능을 보여 왔지만, 신경망 내부의 반복되는 샘플링에 의한 동작 부정확하고 불확실한 경계를 갖는 고질적인 문제가 있다. 반면에 전통적인 영상처리 기법을 이용한 영상 분할 방법은 객체가 뚜렷한 윤곽을 갖으나 의미 단위 분할이 어려운 단점이 있다. 본 논문에서는 영상처리 기법과 심층 신경망을 모두 활용해 의미 단위 분할이 가능하며 더욱 정확한 윤곽을 갖는 영상 분할 방법을 제안한다. 공공 데이터 셋을 이용한 실험에서 제안하는 영상 분할 방법은 심층 신경망 기반 영상 분할기법의 결과를 효율적으로 개선함을 보여준다.

A hierarchical semantic segmentation framework for computer vision-based bridge damage detection

Jingxiao Liu,Yujie Wei,Bingqing Chen,Hae Young Noh 국제구조공학회 2023 Smart Structures and Systems, An International Jou Vol.31 No.4

Computer vision-based damage detection enables non-contact, efficient and low-cost bridge health monitoring, which reduces the need for labor-intensive manual inspection or that for a large number of on-site sensing instruments. By leveraging recent semantic segmentation approaches, we can detect regions of critical structural components and identify damages at pixel level on images. However, existing methods perform poorly when detecting small and thin damages (e.g., cracks); the problem is exacerbated by imbalanced samples. To this end, we incorporate domain knowledge to introduce a hierarchical semantic segmentation framework that imposes a hierarchical semantic relationship between component categories and damage types. For instance, certain types of concrete cracks are only present on bridge columns, and therefore the noncolumn region may be masked out when detecting such damages. In this way, the damage detection model focuses on extracting features from relevant structural components and avoid those from irrelevant regions. We also utilize multi-scale augmentation to preserve contextual information of each image, without losing the ability to handle small and/or thin damages. In addition, our framework employs an importance sampling, where images with rare components are sampled more often, to address sample imbalance. We evaluated our framework on a public synthetic dataset that consists of 2,000 railway bridges. Our framework achieves a 0.836 mean intersection over union (IoU) for structural component segmentation and a 0.483 mean IoU for damage segmentation. Our results have in total 5% and 18% improvements for the structural component segmentation and damage segmentation tasks, respectively, compared to the best-performing baseline model.

Ensemble-based deep learning for autonomous bridge component and damage segmentation leveraging Nested Reg-UNet

Rih-Teng Wu,Abhishek Subedi,Wen Tang,Tarutal Ghosh Mondal,Mohammad R. Jahanshahi 국제구조공학회 2023 Smart Structures and Systems, An International Jou Vol.31 No.4

Computer vision-based damage detection enables non-contact, efficient and low-cost bridge health monitoring, which reduces the need for labor-intensive manual inspection or that for a large number of on-site sensing instruments. By leveraging recent semantic segmentation approaches, we can detect regions of critical structural components and identify damages at pixel level on images. However, existing methods perform poorly when detecting small and thin damages (e.g., cracks); the problem is exacerbated by imbalanced samples. To this end, we incorporate domain knowledge to introduce a hierarchical semantic segmentation framework that imposes a hierarchical semantic relationship between component categories and damage types. For instance, certain types of concrete cracks are only present on bridge columns, and therefore the noncolumn region may be masked out when detecting such damages. In this way, the damage detection model focuses on extracting features from relevant structural components and avoid those from irrelevant regions. We also utilize multi-scale augmentation to preserve contextual information of each image, without losing the ability to handle small and/or thin damages. In addition, our framework employs an importance sampling, where images with rare components are sampled more often, to address sample imbalance. We evaluated our framework on a public synthetic dataset that consists of 2,000 railway bridges. Our framework achieves a 0.836 mean intersection over union (IoU) for structural component segmentation and a 0.483 mean IoU for damage segmentation. Our results have in total 5% and 18% improvements for the structural component segmentation and damage segmentation tasks, respectively, compared to the best-performing baseline model.