Generative Adversarial Networks: A Literature Review

( Jieren Cheng ),( Yue Yang ),( Xiangyan Tang ),( Naixue Xiong ),( Yuan Zhang ),( Feifei Lei ) 한국인터넷정보학회 2020 KSII Transactions on Internet and Information Syst Vol.14 No.12

The Generative Adversarial Networks, as one of the most creative deep learning models in recent years, has achieved great success in computer vision and natural language processing. It uses the game theory to generate the best sample in generator and discriminator. Recently, many deep learning models have been applied to the security field. Along with the idea of “generative” and “adversarial”, researchers are trying to apply Generative Adversarial Networks to the security field. This paper presents the development of Generative Adversarial Networks. We review traditional generation models and typical Generative Adversarial Networks models, analyze the application of their models in natural language processing and computer vision. To emphasize that Generative Adversarial Networks models are feasible to be used in security, we separately review the contributions that their defenses in information security, cyber security and artificial intelligence security. Finally, drawing on the reviewed literature, we provide a broader outlook of this research direction.

High Representation based GAN defense for Adversarial Attack

Richard Evan Sutanto,이석호 한국인터넷방송통신학회 2019 Journal of Advanced Smart Convergence Vol.8 No.1

These days, there are many applications using neural networks as parts of their system. On the other hand, adversarial examples have become an important issue concerining the security of neural networks. A classifier in neural networks can be fooled and make it miss-classified by adversarial examples. There are many research to encounter adversarial examples by using denoising methods. Some of them using GAN (Generative Adversarial Network) in order to remove adversarial noise from input images. By producing an image from generator network that is close enough to the original clean image, the adversarial examples effects can be reduced. However, there is a chance when adversarial noise can survive the approximation process because it is not like a normal noise. In this chance, we propose a research that utilizes high-level representation in the classifier by combining GAN network with a trained U-Net network. This approach focuses on minimizing the loss function on high representation terms, in order to minimize the difference between the high representation level of the clean data and the approximated output of the noisy data in the training dataset. Furthermore, the generated output is checked whether it shows minimum error compared to true label or not. U-Net network is trained with true label to make sure the generated output gives minimum error in the end. At last, the remaining adversarial noise that still exist after low-level approximation can be removed with the U-Net, because of the minimization on high representation terms.

High Representation based GAN defense for Adversarial Attack

Sutanto, Richard Evan,Lee, Suk Ho The Institute of Internet 2019 Journal of Advanced Smart Convergence Vol.8 No.1

These days, there are many applications using neural networks as parts of their system. On the other hand, adversarial examples have become an important issue concerining the security of neural networks. A classifier in neural networks can be fooled and make it miss-classified by adversarial examples. There are many research to encounter adversarial examples by using denoising methods. Some of them using GAN (Generative Adversarial Network) in order to remove adversarial noise from input images. By producing an image from generator network that is close enough to the original clean image, the adversarial examples effects can be reduced. However, there is a chance when adversarial noise can survive the approximation process because it is not like a normal noise. In this chance, we propose a research that utilizes high-level representation in the classifier by combining GAN network with a trained U-Net network. This approach focuses on minimizing the loss function on high representation terms, in order to minimize the difference between the high representation level of the clean data and the approximated output of the noisy data in the training dataset. Furthermore, the generated output is checked whether it shows minimum error compared to true label or not. U-Net network is trained with true label to make sure the generated output gives minimum error in the end. At last, the remaining adversarial noise that still exist after low-level approximation can be removed with the U-Net, because of the minimization on high representation terms.

Generative Adversarial Network을 이용한 한복 디자인 DiscoGAN, CycleGAN, Munit을 중심으로

정유진(Jeong, Yoojin),김경철(Kim, Kyoung Chul),손채봉(Sohn, Chae-Bong) 한국디자인리서치학회 2019 한국디자인리서치 Vol.4 No.3

최근 패션 디자인 분야에서 인공지능을 사용하는 연구가 지속적으로 이루어지고 있다. 그 중 생성 알고리 즘을 적용한 패션 디자인은 2000년대 중반부터 나타나기 시작했다. 생성 알고리즘 중 Generative adversarial network(GAN)은 생성 모델과 판별 모델이 경쟁적으로 학습하면서 실제와 유사한 결과를 만들어내는 방법이다. 본 논문은 GAN 알고리즘으로 윤곽 이미지로부터 한복 이미지를 생성하는 Style transfer 방법을 통해 한복을 디자인한다. Style transfer는 형태는 크게 변하지 않으며 스타일만 변화시키 는 것으로 형태 변화는 크지 않지만 다양한 디자인이 존재하는 한복에 적합하다. 본 논문에서는 Style transfer를 적용하기 위한 한복 이미지와 윤곽 이미지 데이터 셋을 구현하였다. 그 후 대표적인 Style transfer 방법인 DiscoGAN, CycleGAN, 그리고 Muint을 활용해 윤곽 이미지에서 한복이미지를 생성하는 방법과 그 결과를 분석했다. 결과적으로 세 방법 모두 색상 영역과 윤곽 영역 사이의 변환을 학습함으로써 윤곽 이미지가 주어졌을 때 새로운 한복 이미지를 생성해냈다. 또한 기본적인 한복 디자인뿐만 아니라 새로운 한복 무늬나 색의 변화가 있는 옷고름과 같은 창의적인 한복 디자인을 얻을 수 있었다. 이를 통해 인공지능을 사용한 한복 디자인이 가능함과 앞으로의 개발 가능성을 보여준다. Recently, there has been continuous research in fashion design using artificial intelligence. Among them, fashion designs using generation algorithms began to appear in the mid-2000s. Among generation algorithms, Generative Adversarial Network (GAN) is a method that produces plausible samples as generation models and discriminant models competitively trained. In this paper, style transfer methods were used to create Hanbok images based on contour images of Hanbok with GAN algorithm. Style transfer is a suitable way to create hanbok with a variety of designs but no large changes of shape. In this paper, we built our own color and contour images of hanbok dataset for applying style transfer. After that, we analyzed methods and results of design using DiscoGAN, CycleGAN, and Muint, which are representative style transfer methods. As a result, all three methods learned the transformation between the color domain and the contour domain to create a new hanbok image given the contour image. In addition to the basic hanbok design, it designed creative hanbok with new patterns and color changing tie. This paper demonstrates that it is possible to design hanbok using artificial intelligence and future development possibilities.

오토인코더와 다중 레이블 분류기의 사전학습을 통한 GAN 학습 안정성 향상 및 생성 영상 제어

김지수,지준,오희석 대한전자공학회 2024 전자공학회논문지 Vol.61 No.2

적대적 생성 네트워크(Generative Adversarial Network)는 모든 픽셀에 대해 생성망과 판별망이 적대적으로 동작해야 하는 특성상 고해상도 이미지를 생성하기 어렵다는 문제와, 적대적 훈련 과정에 의해 데이터셋 분포 학습이 불안정하다는 문제를 동시에 가지고 있다. 데이터셋 분포 학습 과정에서 영상 제어를 통한 결과물을 생성하기 위해 다양한 기존 연구들이 생성망의 입력인 잠재벡터에 조건을 결합하는 방식으로 영상을 제어한다. 그러나 불안정한 GAN의 특성상 결합한 조건에 부합하는 고품질의 이미지를 생성하는데 한계가 있다. 비전 분야에서 기 학습된 특징 매핑 신경망을 도입해 하위 작업에 전이 학습하여 사용하는 접근은 잘 알려진 학습의 효과를 높여주는 방법이다. 본 논문에서는 전이 학습을 응용하여 생성자의 표현력을 향상시킴과 동시에 학습의 안정성을 도모하고자 하였으며, 생성망의 입력인 잠재벡터에 조건을 결합한 조건 벡터에 부합하는 고해상도 이미지를 생성하고자 하였다. 영상을 저차원의 은닉 공간으로의 임베딩 후 다시 복원하는 일련의 과정을 인코더-디코더를 통해 사전 훈련하였고, 사전 훈련 과정에서 저차원의 은닉 공간으로 임베딩된 벡터를 분류기를 통해 예측한 레이블과 입력 레이블의 오차를 줄여나가는 방향으로 학습시켰다. 고차원의 영상 공간으로 복원하는 네트워크를 생성망의 초기 가중치로 설정하였으며, 저차원의 은닉 공간으로 임베딩하는 네트워크를 생성망 출력을 입력으로 받는 네트워크에 고정 가중치로 설정하여 최종적으로 향상된 학습 안정성과 컨디셔닝 제어를 가능하게 하는 모델인 PSC-GAN (Pre-weighted Stabilized Conditioning Generative Adversarial Network)을 제안함으로써, 동일 조건에서 기존 GAN 학습 시 비교군 기준 대비 정량적으로 10% 가까운 성능 향상이 이루어짐을 확인하였다. Generative Adversarial Networks (GANs) are generative models that face the challenge of sampling high-resolution images due to the generator's hostility towards the discriminator in a pixel-wise manner, as well as the instability of the dataset distribution estimation arising from adversarial training. To generate images that satisfy certain conditions, previous studies have embedded conditions into latent vectors, which are then fed into the generator. However, due to GANs’ training instability, there is a limit to generating images having plausible quality with controlled conditions. One solution is to utilize a pre-trained feature mapper through transfer learning to increase the efficiency of learning. In this paper, we apply transfer learning to improve the fidelity of the generated image and promote training stability, ultimately generating high-resolution images that meet the combined condition and latent representation. We pre-trained the encoder-decoder for embedding an image into a low-dimensional latent space and reconstructing it into a spatial domain, learning the latent space by reducing the errors between the predictions and input labels through the classifier. The weights of the generator were initialized by the pre-trained decoder that reconstructs the image context, while an encoder that embeds an image into latent space was set as a frozen classifier that receives the generator's output. As a result, we verified that retraining the PSC-GAN (Pre-weighted Stabilized Conditioning Generative Adversarial Network) with improved training stability and controllability under the same conditions led to about 10% quantitative improvement compared to previous studies.

딥러닝 기반 교량 손상추정을 위한 Generative Adversarial Network를 이용한 가속도 데이터 생성 모델

이강혁,신도형 한국BIM학회 2019 KIBIM Magazine Vol.9 No.1

Maintenance of aging structures has attracted societal attention. Maintenance of the aging structure can be efficiently performed with a digital twin. In order to maintain the structure based on the digital twin, it is required to accurately detect the damage of the structure. Meanwhile, deep learning-based damage detection approaches have shown good performance for detecting damage of structures. However, in order to develop such deep learning-based damage detection approaches, it is necessary to use a large number of data before and after damage, but there is a problem that the amount of data before and after the damage is unbalanced in reality. In order to solve this problem, this study proposed a method based on Generative adversarial network, one of Generative Model, for generating acceleration data usually used for damage detection approaches. As results, it is confirmed that the acceleration data generated by the GAN has a very similar pattern to the acceleration generated by the simulation with structural analysis software. These results show that not only the pattern of the macroscopic data but also the frequency domain of the acceleration data can be reproduced. Therefore, these findings show that the GAN model can analyze complex acceleration data on its own, and it is thought that this data can help training of the deep learning-based damage detection approaches.

Counterfactual image generation by disentangling data attributes with deep generative models

Lim Jieon,Joo Weonyoung 한국통계학회 2023 Communications for statistical applications and me Vol.30 No.6

Deep generative models target to infer the underlying true data distribution, and it leads to a huge success in generating fake-but-realistic data. Regarding such a perspective, the data attributes can be a crucial factor in the data generation process since non-existent counterfactual samples can be generated by altering certain factors. For example, we can generate new portrait images by flipping the gender attribute or altering the hair color attributes. This paper proposes counterfactual disentangled variational autoencoder generative adversarial networks (CDVAE-GAN), specialized for data attribute level counterfactual data generation. The structure of the proposed CDVAE-GAN consists of variational autoencoders and generative adversarial networks. Specifically, we adopt a Gaussian variational autoencoder to extract low-dimensional disentangled data features and auxiliary Bernoulli latent variables to model the data attributes separately. Also, we utilize a generative adversarial network to generate data with high fidelity. By enjoying the benefits of the variational autoencoder with the additional Bernoulli latent variables and the generative adversarial network, the proposed CDVAE-GAN can control the data attributes, and it enables producing counterfactual data. Our experimental result on the CelebA dataset qualitatively shows that the generated samples from CDVAE-GAN are realistic. Also, the quantitative results support that the proposed model can produce data that can deceive other machine learning classifiers with the altered data attributes.

Infrared and visible image fusion based on Laplacian pyramid and generative adversarial network

( Juan Wang ),( Cong Ke ),( Minghu Wu ),( Min Liu ),( Chunyan Zeng ) 한국인터넷정보학회 2021 KSII Transactions on Internet and Information Syst Vol.15 No.5

An image with infrared features and visible details is obtained by processing infrared and visible images. In this paper, a fusion method based on Laplacian pyramid and generative adversarial network is proposed to obtain high quality fusion images, termed as Laplacian-GAN. Firstly, the base and detail layers are obtained by decomposing the source images. Secondly, we utilize the Laplacian pyramid-based method to fuse these base layers to obtain more information of the base layer. Thirdly, the detail part is fused by a generative adversarial network. In addition, generative adversarial network avoids the manual design complicated fusion rules. Finally, the fused base layer and fused detail layer are reconstructed to obtain the fused image. Experimental results demonstrate that the proposed method can obtain state-of-the-art fusion performance in both visual quality and objective assessment. In terms of visual observation, the fusion image obtained by Laplacian-GAN algorithm in this paper is clearer in detail. At the same time, in the six metrics of MI, AG, EI, MS_SSIM, Q_abf and SCD, the algorithm presented in this paper has improved by 0.62%, 7.10%, 14.53%, 12.18%, 34.33% and 12.23%, respectively, compared with the best of the other three algorithms. abfQ

고해상도 지도 생성을 위해서 ERF를 고려한 GAN

이기언 한국정보과학회 2019 정보과학회논문지 Vol.46 No.2

The paper proposes a network structure for a generative adversarial network (GAN) suitable for high resolution image transformation. For analysis of the resolution classification relation necessary for high resolution image conversion, the effective size of the receptive fields of each encoder is calculated and new connection imbalance fields defined. We can reduce the total number of layers by connecting the encoder and decoder to the patch size, we reduce the total number of layers and the appropriate effective receptive fields and parameter usability confirmed through experiments. To solve the problem of simultaneously providing resolution and classification in high resolution image conversion, a network structure capable of converting high resolution satellite images is suggested experimentally. Additionally, the validity of the network structure that simultaneously improves the resolution and classification is confirmed by comparing and analyzing the receptive fields of the proposed network and the existing network’s receptive fields. The proposed network is then quantitatively verified by comparing the proposed network with the existing network by use of objective numerical value through SSIM, an image similarity analysis method. 본 논문은 고해상도 이미지 변환에 적합한 GAN(Generative Adversarial Network)의 네트워크 구조를 제안한다. 고해상도 이미지 변환에 필수적인 해상도와 분류 관계를 분석하기 위해 각 인코더들의 effective receptive fields의 크기를 계산하고, 새롭게 connection imbalance fields를 정의한다. 인코더와 디코더 간을 patch 단위로 연결하여 전체 층 수를 줄임으로써 적절한 effective receptive fields와 매개변수 사용 가능성을 실험을 통해 확인한다. 고해상도 이미지 변환 시에 해상도와 분류를 동시에 제공하기 어려운 문제를 개선하기 위해 고해상도 위성 사진을 변환할 수 있는 네트워크 구조를 실험적으로 제시한다. 또한 제시된 네트워크와 기존 네트워크의 receptive fields 크기를 비교 분석하여, 해상도와 분류를 동시에 향상시키는 네트워크 구조에 대한 타당성을 확인한다. 그리고, 제시된 네트워크와 기존의 네트워크를 이미지 유사도 분석 방법인 SSIM을 통해서 객관적 수치를 통해 비교함으로써 제안된 구조의 적합성을 정량적으로 검증한다.

자율 이동 로봇의 경로 계획에 적용된 심층 신경망 : 리뷰논문

김정은,서은성,석준희 한국차세대컴퓨팅학회 2022 한국차세대컴퓨팅학회 학술대회 Vol.2022 No.05

최근 군사, 산업 등 다양한 분야에서 자율 이동 로봇이 활용되고 있다. 자율 이동 로봇이 특정 임무를 수행하는 데 있어 경로 계획은 핵심적인 역할을 한다. 오랜 기간 동안 A*, Rapidly exploring Random Tree 등 많은 경로 탐색 알고리즘이 연구되었다. 하지만 과거에 연구된 알고리즘들은 짧은 시간 내에 장애물과 충돌이 일어나지 않는 최단 경로를 찾는 데 애로사항이 존재한다. 이를 해결하기 위해 최근 활발하게 연구되고 있는 심층 신경망을 사용하여 경로를 생성하는 연구 사례들이 늘어나고 있다. 심층 신경망은 비선형 함수를 근사하는데 좋은 결과를 보여주어 경로 계획에 필요한 함수의 근사에 사용될 수 있다. 다양한 딥러닝 모델이 비선형 함수를 근사하지만 특히 예측 모델을 사용하는 것이 아닌 생성적 모델을 사용하는 것으로 원하는 경로를 좀 더 효율적으로 생성할 수 있을 것으로 예측된다. 본 논문에서는 컴퓨터 비전 분야에서 성공적인 결과를 보여준 Generative Adversarial Network를 통해 전역 경로를 이미지로 표시하는 데 성공한 사례를 제시한다. 또한 에이전트의 행동 제어에서 좋은 성과를 보여준 심층 강화학습을 사용해 자율 이동 로봇의 지역 경로 계획에 성공한 연구를 소개한다.