Knowledge distillation of regression neural network

Kang, Myeonginn Sungkyunkwan University 2024 국내박사

최근 다양한 산업 분야에서 인공신경망 적용이 큰 성공을 거두고 있다. 그러나 거대 인공신경망은 많은 양의 계산과 자원을 필요로 하기 때문에 자원의 제약이 있는 환경에서의 활용에 어려움이 있다. 이를 해결하기 위해 거대 인공신경망 모델 (teacher)을 작은 인공신경망 모델 (student)로 압축하는 knowledge distillation 연구가 활발히 진행 중이다. 기존 knowledge distillation 방법들은 teacher network를 학습시키는데 사용했던 학습 데이터셋이 모두 재사용 가능함을 가정한다. 그러나 현실 문제에서는 다양한 제약으로 인해 학습 데이터셋이 항상 온전히 보존되거나 공개되지 않을 수 있다. 이를 해결하기 위해 학습 데이터셋이 일부 사용가능한 상황을 가정한 knowledge distillation 방법들이 제안되었지만 모두 분류 문제에 집중하고있다. 본 논문에서는 회귀 인공신경망에 적용될 수 있는 새로운 knowledge distillation 방법들을 제안한다. 첫째로, 학습 데이터셋 사용이 불가능한 상황에서 회귀 인공신경망에 적용 가능한 data-free knowledge distillation 방법을 제안한다. 거대한 teacher network가 주어졌을 때, generator network를 도입하여 teacher network의 지식을 작은 student network로 전이한다. Generator와 student network는 적대적 학습 방식을 사용해 동시에 학습된다. Generator network는 teacher와 student의 예측 차가 커지도록 하는 인공 데이터 포인트를 생성하도록 학습되고, 반면에 student network는 생성된 인공 데이터 포인트에 대한 teacher와 student의 예측 차를 줄이도록 학습된다. 둘째로, 학습 데이터셋이 일부 재사용 가능한 상황에서 회귀 인공신경망에 적용 가능한 새로운 knowledge distillation 방법인 teacher-student matching (TSM)을 제안한다. TSM은 세 개의 학습 방식을 포함한다: Perturbation-based matching (PM), Adversarial belief matching (ABM), Gradient matching (GM). TSM은 학습 데이터 부족 상황에서 기존의 knowledge distillation 방법의 성능 개선을 위해 추가 적용 방법으로 사용될 수 있다. 마지막으로, 학습 데이터셋 사용 불가능한 상황에서 회귀 인공신경망의 예측 불확실성 정량화를 위한 대리 기법을 제안한다. 이를 위해 첫번째로 제안한 회귀 인공신경망을 위한 data-free knowledge distillation 방법을 활용한다. Data-free knowledge distillation 방법과 추가적인 세 개의 대리 기법을 사용한다: Input perturbation, Gradient norm, MC-dropout, Knowledge distillation. 쿼리 데이터 포인트가 주어졌을 때, 각 대리 기법은 학습 데이터셋 사용 없이 회귀 인공신경망을 사용해 예측 불확실성을 정량화한다. 회귀 벤치마크 데이터셋에 대한 실험을 통해 각 제안 방법의 효과를 확인하였다. Artificial neural networks have been widely used in various industrial fields. For more efficient use of artificial neural networks in environments with limited computing resources, knowledge distillation is actively applied to compress a large neural network (teacher) to a smaller neural network (student). Conventional knowledge distillation requires a training dataset that was used to build the teacher network. However, the training dataset is often not fully accessible in many real-world applications due to some practical issues. To solve this problem, there are existing methods of knowledge distillation with insufficient training data, but they only focus on classification problems. This dissertation proposes novel knowledge distillation methods that can be applied to a regression network. First, we propose data-free knowledge distillation of the regression network. Given a large teacher network, a generator network is adopted to transfer the knowledge in the teacher network to a smaller student network. The generator and student networks are simultaneously trained in an adversarial manner. The generator network is trained to create synthetic data on which the teacher and student networks make different predictions, with the student network being trained to mimic the teacher network's predictions. Second, we propose knowledge distillation of the regression network with insufficient training data, called teacher-student matching (TSM). TSM includes three additional learning objectives that are modifications of existing knowledge distillation methods to make the student better emulate the prediction capability of the teacher: perturbation-based matching (PM), adversarial belief matching (ABM), and gradient matching (GM). TSM can be used as an add-on to any existing knowledge distillation method to improve its effectiveness under severe data insufficiency. Third, we propose a surrogate approach to quantify the prediction uncertainty of the regression network without any training data. To do this, we utilize the data-free knowledge distillation of the regression network. While the original aim of knowledge distillation is to compress the large neural network, we expand the use of knowledge distillation to quantify the prediction uncertainty. Four surrogate measures are introduced: Input perturbation, Gradient norm, MC-dropout, and Knowledge distillation. For a query data point, each surrogate measure can be calculated by using the regression network only to estimate the prediction uncertainty. The effectiveness of the proposed methods is demonstrated through experiments on regression benchmark datasets.

Combining deep learning with domain knowledge for wafer map pattern classification

Kang, Hyungu Sungkyunkwan University 2022 국내석사

Recently, machine learning has been effectively applied in the automation of wafer map pattern classification in semiconductor manufacturing. One conventional approach is to extract handcrafted features from a wafer map and build an off-the-shelf classifier on top of the features. Another approach is to use a convolutional neural network that operates directly on a wafer map. These two approaches have different strengths for different classes of wafer map defect patterns. In this study, we present a hybrid method that leverages the advantages of both approaches to improve the classification accuracy. First, we build two base classifiers using each of the approaches. Then, we build a stacking ensemble that combines the outputs of these base classifiers for the final prediction. The stacking ensemble classifies a wafer map by assigning a larger weight to the output of the superior base classifier with respect to each defect class. We demonstrate the effectiveness of the proposed method using real-world data from a semiconductor manufacturer. 최근 반도체 제조 산업에서 웨이퍼 맵의 불량 범주 분류 자동화를 위해 기계학습이 효과적으로 적용되고 있다. 기존의 접근법 중 하나는 전문가 지식을 활용하여 웨이퍼 맵에서 수동으로 특성을 추출한 후, 이 특성을 이용하여 기성 분류 모델을 학습시키는 방법이다. 또 다른 접근법은 웨이퍼 맵 자체를 입력으로 사용하여 합성곱 신경망으로 학습시키는 딥러닝 방법이다. 본 연구에서는 분류 정확도를 높이기 위해 전문가 지식과 딥러닝을 결합한 하이브리드 방법을 제안한다. 먼저 수동 특성 추출한 뒤 기성 분류 모델을 학습하는 방법과 합성곱 신경망으로 학습하는 방법을 이용하여 두 개의 단일 모델을 만들었다. 그 후 단일 모델들의 예측 결과값을 이용해 스태킹 앙상블 모델을 구축하였다. 단일 모델들은 특성 추출 방식의 차이로 인해 학습 데이터셋의 크기에 따라 성능 차이가 발생하며 서로 다른 불량 범주에 대해 다른 강점을 갖는다. 따라서 제안 방법은 불량 범주마다 더 우세한 모델의 예측값에 높은 가중치를 부여함으로써 두 접근법의 장점을 모두 끌어올리며 최종적으로 예측 정확도를 향상한다. 실제 반도체 제조 현장의 데이터인 WM-811K 데이터셋을 사용하여 다양한 학습 데이터셋의 크기에 대해 제안 방법의 성능이 단일 모델에 비해 개선됨을 검증하였다.

Characteristics and restrictiveness of rules of origin in the Korea-Australia FTA : an empirical analysis

Kang, Narae Korea University 2017 국내석사

Rules of origin (RoO) are necessary and important in free trade agreements (FTAs), given the fact that their function is to prevent trade deflection. However, with the proliferation of FTAs over the last two decades, diverse RoO among the different FTAs have resulted in increases in the cost of complying with the complex requirement of RoO. In other words, RoO can play a role as trade barriers. Thus, it is critical to find out how demanding RoO are, in order not to limit exporters’ opportunities for more markets. On this ground, this paper analyzed the restrictiveness of RoO, which can be hidden protection, with the example of the bilateral FTA between Korea and Australia, using a method proposed by Estevadeordal (2000). It revealed that the restrictiveness index of the Korea-Australia FTA is 4.26, lower than those of the Korea-China FTA (4.43), the Korea-EFTA FTA (4.53), the Korea-ASEAN FTA (4.59), and the Korea-Chile FTA (4.82). This low restrictiveness index of the Korea-Australia FTA can be explained mainly by the complementary industrial and trade structure and significant amount of trade volume between the two countries. Then, examining restrictiveness of RoO for nineteen sectors, it is found that the agricultural and animal sector is the most restrictive among all the sectors, whereas the chemical and electrical equipment sectors are less restrictive. In addition, the analysis has shown that the restrictiveness of RoO in major five sectors in the Korea-Australia FTA lies between those of the China-Australia FTA and the Japan-Australia FTA. Given the results of this research, even though RoO in the Korea-Australia FTA are less restrictive than those of Korea’s other FTAs, Korea should adopt a more strategic approach to trade policy, considering the restrictiveness of RoO and Korea’s position in the Australian market vis-à-vis China and Japan. Furthermore, the Korean government needs to review these factors for renegotiation of the Korea-Australia FTA in the future.

(A) study on high speed and low power circuit designs for asynchronized parallel wire link interface of NAND flash memory

Kang, Kyungtae Sungkyunkwan university 2019 국내박사

As higher operating frequency is required in NAND Flash Memory's application, as much more power consumption increases because the techniques to provide the stable data transfer such as DCC (Duty Cycle Corrector), per-pin de-skewing, and pre-emphasis transmitter are adopted. In this thesis paper, a study for minimizing these additional power consumption is presented by using the proposed circuits with maintain its own performance. First of all, the proposed DCC consists of a loop delay chain for edge alignment, and a falling edge modulator to enhance the phase interpolating limit. These features improved the duty offset correction range at high frequency besides low frequency with fast lock time and without degrading the signal integrity of the junction of a phase interpolator. The proposed DCC was fabricated in a TSMC 55nm CMOS technology with 1 V supply voltage, the area occupied 0.0186mm2. The measured results show that the duty cycle error of the output clock was adjusted to less than 2% when the duty cycle ratio of the input clock was changed from 80% to 20% at 1 GHz, and the lock cycle consumed only 5 cycles. At 1 GHz, the power consumption was 2.09mW and the peak-to-peak jitter was measured at 12.53 ps. Second of all, it presents an open-loop per pin skew compensation with lock fault detection. The proposed circuit employs an open-loop reference selector, a 2-stage open-loop delay lock method which is separated by a coarse and fine lock for fast lock-in time, and a lock fault detecting scheme to prevent lock fault by dead-zone of samplers. We also applied a unidirectional scan method ahead the fine lock stage to minimize pin-to-pin skew errors after calibration. The circuit was fabricated with TSMC 55nm CMOS technology with a 1V supply voltage and an area of 0.0036mm2 for one de-skewing module. The measured result shows that the skew error at 1GHz operation was reduced to less than 6 ps after skew calibration when the skew between IO pins was 230 ps, and the lock-in time was 11 clock cycles. Third of all, by modulating a post-cursor signal in 2-tap pre-emphasis transmitter, dissipation current of data transfer is reduced by 6% at 3.2Gbps, compared to conventional 2-tap pre-emphasis transmitter. This modulated post-cursor in this paper serves to reduce this static current dissipation period caused by summing two different polarity of the main and post cursor in a transmitter output driver stage. The proposed power-efficient pre-emphasis transmitter was designed in 1V supply using TSMC 55nm process and simulated under the channel environment of a multiple-stacked NAND Flash Memory.

The Korean struggle for internationalidentity in the foreground of the shufeldt negotiation, 1866-1882

Kang UMI 1990 국내박사

Into the unknown part of the human limb dynamics identification using a robot : the robot inertial and flexible dynamics’ effects and their treatment

Hyunah Kang Ulsan National Institute of Science and Technology 2023 국내박사

Cerebral vasoconstriction after carotid artery stenting associated with isolated cerebral circulation

Kang, Chul Hoo 제주대학교 대학원 2020 국내석사

배경 및 목적 경동맥 스텐트 삽입술은 경동맥 협착증의 치료로 널리 행해지고 있다. 경동맥 스텐트 삽입술 후 많은 변화가 일어날 수 있는데, 이 중 하나가 뇌혈관 수축이다. 이러한 뇌혈관 수축은 뇌졸중과 같은 시술 후 합병증과 연관이 있기 때문에 유의 깊게 관찰하여야 한다. 이 연구에서는 내경동맥이 거의 막힌 환자에서 경동맥 스텐트 삽입술 직후의 뇌혈관 조영술을 통해 뇌혈관 수축을 관찰하고자 하였다. 대상과 방법 2008년 12월부터 2019년 5월까지 스텐트 삽입술을 시행한 314명 환자들의 임상자료와 영상자료를 후향적으로 분석하였다. 이 중 30명에서 내경동맥이 거의 막혀 있었다. 두 명의 영상의학과 전문의가 시술 전 자기공명 혈관영상을 통해 뇌혈류의 고립이 있는지와 내경동맥 스텐트 삽입술 직후의 뇌혈관 조영술을 통해 뇌혈관 수축이 있는지를 판단하였다. 뇌혈류의 고립은 자기공명 혈관영상에서 내경동맥과 중간대뇌동맥의 신호 강도가 감소되어 있으면서, 동측 앞대뇌동맥의 교통이전부분과 뒤교통동맥이 보이지 않는 경우로 정의하였고, 뇌혈관 수축은 시술 전, 후의 뇌혈관 조영술 결과를 비교하여 viii 시술 전보다 시술 후에 뇌혈관이 좁아진 경우로 정의하였다. 결과 내경동맥의 근접 폐색이 있는 30명의 임상자료와 영상자료를 분석하였고, 남성이 27명(90.0%)이었으며, 환자들의 평균 연령은 69.0세였다. 이 중, 11명(36.7%)에서 두개내 뇌혈관 수축을 보였고, 나머지 19명은 뇌혈관 수축을 보이지 않았다. 두 군 사이에 동반질환, 경동맥 협착으로 인한 증상 유무, 복용하던 항혈소판제의 종류, 평균 시술 시간, 초기 NIHSS (National Institutes of Health Stroke Scale) 점수와 mRS (modified Rankin Scale) 점수에 있어 통계적으로 유의한 차이는 보이지 않았다. 그러나 뇌혈관 수축은 뇌혈류 고립이 있는 환자에서 통계적으로 유의하게 높은 빈도로 발생하였다. (뇌혈류 고립이 있는 군 64.2%, 뇌혈관 고립이 없는 군 12.5%; p<0.05) 뇌혈관 수축이 발생한 11명의 환자에서 두통이나 다른 신경학적 증상은 없었다. 결론 내경동맥이 거의 막힌 환자에서 경동맥 스텐트 삽입술 이후 뇌혈관 수축은 약 1/3의 환자에서 발생하며, 뇌혈류의 고립이 있는 환자에서 유의하게 높은 빈도로 발생한다. 이러한 뇌혈관 수축의 임상적 영향을 평가하기 위해 향후 대규모 연구가 필요하다. Background and Purpose Carotid artery stenting (CAS) is widely performed for treatment of carotid stenosis. Many changes can occur after carotid artery stenting, and one of which is cerebral vasoconstriction. Cerebral vasoconstriction should be observed carefully, because it is associated with post-procedural complication such as stroke. The purpose of this study is to present our observation on cerebral vasoconstriction in the ipsilateral anterior circulation during immediate post-stenting angiography in patients with near total occlusion (NTO) of proximal internal carotid artery. (ICA) Materials and Methods We retrospectively reviewed 314 patients’ data from December 2008 to May 2019. There were 30 patients with carotid NTO. Two neuroradiologists reviewed time-of-flight (TOF) magnetic resonance angiography (MRA) to evaluate the presence of isolated circulation, and reviewed the final cerebral angiographic finding of CAS to evaluate the presence of cerebral vasoconstriction. Isolated circulation was defined as 1) signal intensity drop of the ipsilateral middle cerebral artery/ICA territory and 2) absence of ipsilateral A1 segment and posterior communicating artery when evaluated on TOF MRA. Cerebral vasoconstriction was defined as the narrowing of cerebral vessels on post-stenting angiography compared to pre-stenting angiography. Results A total of 30 patients with NTO were analyzed. 11 patients showed vasoconstriction in the treated territory, and 19 patients did not show significant vasoconstriction after CAS. There were no statistically significant differences in comorbidity, frequency of symptomatic lesions, antiplatelet medication, mean procedure time, and initial National Institutes of Health Stroke Scale and baseline modified Rankin Scale scores between the two groups. However, cerebral vasoconstriction is more likely to happen in patients with isolated territory from the contralateral anterior and posterior circulation (64.2% in the isolated territory group and 12.5% in the not-isolated territory group; p<0.05). No headache or neurologic deficit was noted in all 11 patients with cerebral vasoconstriction. Conclusions Cerebral vasoconstriction after CAS occurs in about one third of patients with NTO of proximal ICA, and it occurs more frequently in patients with isolation of the cerebral circulation. A large-scale study is necessary to assess the clinical implication of cerebral vasoconstriction after CAS.

Study of protein cage nanoparticles for biomedical application

Youngji Kang Graduate School of UNIST 2016 국내박사

Biomedical science has been greatly advanced and recognized as an important area to improve human healthcare. Recent advances in nanotechnology have been appreciated for developing biomedical sciences with its potential of generating unique materials, devices and systems utilizing phenomena and properties of nanoscale substances. The terminology, nanobiotechnology, is the combination of biotechnology and nanotechnology, and has been recognized as future promising technology. Nanobiotechnology usually represents a range of material size from 1 nm to 100 nm. Nanoscale materials have been widely utilized to observe physical, chemical and biological properties of substances. There are diverse nanomaterials including quantum dots (QDs), liposomes, dendrimers, virus-like particles (VLPs), and protein cages, utilized for biomedical applications. Biomedical applications can be generally classified into cell targeting, drug delivery, bioimaging, and vaccine development. In order to utilize nanoscale substances for such biomedical applications, their surface can be easily modifiable and they should contain optimal solubility and stability in aqueous media, without any side effects on biological properties and functionalities. Among the number of nanomaterials, protein cages such as ferritin, heat shock protein and Lumazine synthase have been highlighted with their advantages of biocompatibility, non-toxic, low cost for production and biodegradability, and applied for various biomedical sciences. Protein cages contain a hollow internal space which can encapsulate small molecules like fluorescence dyes or probes as a detector, and have an external surface which can be modifiable by using bioconjugation and azide-alkyne click chemistry. In this study, we focus on developing diverse protein cages for biomedical applications, particularly for cell targeting system.

Development of Target-Ligand Switchable Lactate Oxidase Systems for Cancer Therapy

Yujin Kang Ulsan National Institute of Science and Technology 2023 국내석사

The aggressive tumor formation often causes excessive anaerobic glycolysis leading to the massive production of lactate and its accumulation to the tumor microenvironment (TME). Therefore, it is important to control the lactate concentration in TME to properly modulate surrounding tumor cells and suppress tumor growth. Lactate Oxidase (LOX) is a tetrameric enzyme converting lactate to pyruvate and H2O2 in the presence of oxygen and target-ligand (vSIRPα, EGFRAfb, HER2Afb) is a ligand that interacts with receptor frequently overexpressed on the surface of cancer cells. To control locally accumulated lactate properly, LOX and target-ligand were used as a potential therapeutic enzyme and a tumor cell targeting ligand, respectively, and LOX/ target-ligand conjugates were constructed through a SpyTag/SpyCatcher protein ligation system. LOX/target-ligand selectively bound to the specific receptor overexpressing tumor cells and effectively consumed lactate produced by tumor cells generating adequate amounts of H2O2, which induce drastic necrotic tumor cell death. Local treatments of B16-F10 tumor-bearing mice with LOX/vSIRPα significantly suppressed tumor growth without any severe side effects. Tumor-targeting vSIRPα may allow longer retention of LOX onto tumor sites effectively consuming surrounding lactate in TME and locally generating adequate amounts of H2O2 to suppress tumor growth. The approach controlling the local lactate concentration and H2O2 in TME using LOX and various target-ligand would offer new opportunities for developing enzyme/target-ligand conjugate-based therapeutic tools for tumor treatment.

The making of Confucian societies in Tokugawa Japan and Yi Korea : a comparative analysis of the behavior pattepns in accepting the foreign ideology, Neo-Confucianism

Kang UMI 1990 국내박사