앙상블 경험적 모드 분해를 이용한 수질자료의 이상치 탐색

박상수,박노석,김성수,조귀래,윤석민 대한환경공학회 2021 대한환경공학회지 Vol.43 No.3

Objectives:This study was conducted to propose a new methodology for efficiently identifying and removing various outliers that occur in data collected through automated water quality monitoring systems. In the present study, water temperature data were collected from domestic G_water supply system, and the performance of the proposed methodology was tested for water temperature data collected from domestic G_water supply system. Methods:We applied the following analytical procedure to identify outliers in the water quality data: First, a normality test was performed on the collected data. If normality condition was satisfied, the Z-score was used. However, if the normality condition was not satisfied, outliers were identified using the quartile, and the limitations of the existing methodology were analyzed. Second, we decomposed the intrinsic mode function using empirical mode decomposition and ensemble empirical mode decomposition for the collected data, and then considered the occurrence of modal mixing. Finally, a group of intrinsic mode functions was selected using statistical characteristics to identify outliers. In addition, the performance of the method was verified after removing and interpolating outliers using regression analysis and Cook’s distance. Results and Discussion:In the case of water temperature data, as normality condition was not satisfied, outlier identification was carried out by applying the modified quartile method. It was confirmed that outliers distributed within the seasonal component could not be identified at all. In the case of empirical mode decomposition, modal mixing occurred because of the effect of outliers. However, in the case of the ensemble empirical mode decomposition, modal mixing was resolved and the distinct seasonal components were decomposed as intrinsic mode functions. The intrinsic mode functions were synthesized, which showed statistical correlation with the raw water temperature data. As a result of developing a regression model using the synthesized intrinsic mode functions and raw water temperature data and performing outlier search based on Cook’s distances, we concluded that various outliers distributed within the seasonal component could be effectively identified. Conclusions:Considering that satisfactory results could be derived from statistical analysis of the data collected from the automated water quality monitoring system, it can be concluded that outlier identification procedures are essential. However, in the case of the conventional univariate outlier search method, it is apparent that the outlier search performance is significantly poor for data with strong inherent variability, and the interpolation method for the searched outlier cannot be performed. Conversely, the outlier identification method based on ensemble empirical mode decomposition and regression analysis proposed in this study shows excellent discrimination performance for outliers distributed in data with strong inherent variability. Moreover, this method has the advantage of reducing the analyst’s dependence on subjective judgment by presenting statistical cutoff criteria. An additional advantage of the method is that data can be interpolated after removing outliers using intrinsic mode functions. Therefore, the outlier search and interpolation method proposed in this study is expected to have greater applicability as a more effective analysis tool compared to the existing univariate outlier search method. 목적:본 연구는 국내 상수도 자동수질측정망을 통해 수집되는 자료에서 발생 가능한 다양한 이상치들을 효율적으로 탐색 및 제거 위한 방법론을 제안하기 위해 수행되었다. 이를 위해 국내 G_정수장으로부터 수온자료를 수집하였으며, 수집된 자료를 대상으로 이상치 방법론에 따른 적용 효과를 검정하였다. 방법:본 연구에서 수질자료의 이상치 탐색을 위해 적용한 분석 절차는 다음과 같다. 첫째, 수집된 수온자료에 대해 정규성 검정을 수행하고 정규성을 만족하는 경우 Z-score, 정규성을 만족하지 않는 경우 사분위수를 활용하여 이상치를 탐색하고 기존 방법론의 한계점에 대해 분석한다. 둘째, 수온자료에 대해 경험적 모드 분해 및 앙상블 경험적 모드 분해를 활용하여 고유진동함수들을 분해한 후 모드 믹싱에 발생에 대해 고찰한다. 최종적으로 고유진동함수들의 통계적 특성치를 활용해 이상치를 식별할 기준 고유진동함수 집단을 선별한 후 회귀분석과 Cook 통계량의 절사 기준을 활용해 이상치를 제거 및 보간 후 그 성능을 검증한다. 결과 및 토의:수온자료의 경우 정규성을 만족하지 못하며, 수정 사분위 방법을 적용하여 이상치 탐색을 수행한 결과 계절 성분 내에 분포하는 이상치들은 전혀 식별할 수 없다는 결과를 확인하였다. 경험적 모드 분해의 경우 이상치들의 효과로 인해 모드 믹싱 현상이 발생하였으나, 앙상블 경험적 모드 분해에서는 모드 믹싱이 해결되어 뚜렷한 계절 성분이 고유진동함수로서 분해되는 것으로 나타났다. 그리고 앙상블 모드 분해로부터 구해진 고유진동함수 중 원시 수온자료와 통계적 관계성이 높은 신호들을 합성하였다. 합성된 고유진동함수와 원시 수온자료를 활용해 회귀 모형을 개발하고, Cook 통계량 근간으로 이상치 탐색을 수행한 결과 계절 성분 내에 분포하는 다양한 이상치들을 효과적으로 탐색할 수 있는 것으로 분석되었다. 결론:상수도 자동수질측정망을 통해 수집되는 자료들로부터 합리적인 통계분석 결과를 도출하기 위한 과정에서 이상치 탐색 작업은 필수적이라고 할 수 있다. 하지만 기존의 단변량 이상치 탐색 기법의 경우 고유 변동성이 강하게 분포하는 자료에 대해 이상치 탐색 성능이 현저히 떨어지며, 탐색된 이상치에 대한 내삽 방안도 제시하지 못한다는 한계가 명확하다. 반면, 본 연구에서 제시한 앙상블 경험적 모드 분해 및 회귀분석 기반의 이상치 탐색 방법은 고유 변동성이 강한 자료 내에 분포하는 이상치들에 대한 식별 성능이 뛰어나며, 통계적 절사 기준을 제시함에 따라 분석자의 주관적 판단을 최소화 할 수 있는 장점이 있다. 또한 앙상블 경험적 모드 분해 분석으로부터 구해진 고유진동함수들을 이용해 이상치 제거 후 자료 보간이 가능하다는 장점이 있다. 따라서 기존의 단변량 이상치 탐색 기법의 적용성에 대한 한계를 고려할 때 본 연구에서 제시한 이상치 탐색 및 보간 방안은 보다 효과적인 분석 도구로서 적용 가능할 것으로 기대된다.

환경부 8일 유량,수질 자료를 이용한 SWAT 자동보정 모듈 개선 및 적용 평가

강현우 ( Hyun Woo Kang ),류지철 ( Ji Chul Ryu ),강형식 ( Hyung Sik Kang ),최재완 ( Jae Wan Choi ),문종필 ( Jong Pil Moon ),최중대 ( Joong Dae Choi ),임경재 ( Kyoung Jae Lim ) 한국물환경학회 2012 한국물환경학회지 Vol.28 No.2

Soil and Water Assessment Tool (SWAT) model has been widely used in estimation of flow and water quality at various watersheds worldwide, and it has an auto-calibration tool that could calibrate the flow and water quality data automatically from thousands of simulations. However, only continuous measured day flow/water quality data could be used in the current SWAT auto-calibration tool. Therefore, 8-day interval flow and water quality data measured nationwide by Korean Ministry of Environment (MOE) could not be used in SWAT auto-calibration even though long-term flow and water quality data in the Korean Total Maximum Daily Load (TMDL) watersheds available. In this study, current SWAT auto-calibration was modified to calibrate flow and water quality using 8-day interval flow and water quality data. As a result of this study, the Nash and Sutcliffe Efficiency (NSE) values for flow estimation using auto-calibration are 0.77 (calibration period) and 0.68 (validation period), and NSE value for water quality (T-P load) estimation (using the 8-day interval water quality data) is 0.80. The enhanced SWAT auto-calibration could be used in the estimation of continuous flow and water quality data at the outlet of TMDL watersheds and ungaged point of watersheds. In the next study, the enhanced SWAT auto-calibration will be integrated with Web based Load Duration Curve (LDC) system, and it could be suggested as methods of appraisal of TMDL in South Korea.

팔당댐 유역의 데이터 기반 수질 예측 모형 구성을 위한 사전 불확실성 분석

이은정(Eun Jeong Lee),금호준(Ho Jun Keum) 응용생태공학회 2022 Ecology and resilient infrastructure Vol.9 No.1

수질분야에서 물재해 안정성 강화를 위해 과거와 현재의 수질을 분석하여 예측하는 기술을 지속적으로 고도화하는 것이 필요하며 데이터 기반의 예측 모형이 하나의 대안으로 대두되고 있다. 데이터 기반 모형은 복잡하고 광범위한 자료의 양을 기반으로 구축되기 때문에 보다 신뢰도 있는 결과를 얻을 수 있는 입력자료의 조합을 위한 상관관계 분석방법의 적용이 필수적이다. 본 연구에서는 보다 신속하고 정확한 데이터 기반의 수질 예측 모형을 구성하기 위한 선행단계로 Gamma Test를 적용하였다. 먼저 팔당댐의 다양한 수문조건에 따른 해당 유역의 복잡성과 정밀성이 재현된 과거와 현재의 일단위 수질을 최대한 확보하고자 물리적 기반 모형 (HSPF, EFDC)을 구동하였다. 팔당댐 수질예측지점과 팔당댐으로 유입되는 주요 하천의 수질을 대상으로 Gamma Test를 수행한 후 해석결과 (Gamma, Gradient, Standar Error, V-Ratio)를 통해 최적의 자료조합을 선정하는 방법을 제시하였다. 본 연구의 결과는 데이터 기반 모형 구축 시 반복적인 수행과정을 생략하여 시간을 단축하면서 보다 효율적으로 최적의 입력자료를 선정할 수 있는 정량적인 기준을 보여준다. For water quality management, it is necessary to continuously improve the forecasting by analyzing the past water quality, and a Data-driven model is emerging as an alternative. Because the Data-driven model is built based on a wide range of data, it is essential to apply the correlation analysis method for the combination of input variables to obtain more reliable results. In this study, the Gamma Test was applied as a preceding step to build a faster and more accurate data-driven water quality prediction model. First, a physical-based model (HSPF, EFDC) was operated to produce daily water quality reflecting the complexity of the watershed according to various hydrological conditions for Paldang Dam. The Gamma Test was performed on the water quality at the water quality prediction site (Paldangdam2) and major rivers flowing into the Paldang Dam, and the method of selecting the optimal input data combination was presented through the analysis results (Gamma, Gradient, Standar Error, V-Ratio). As a result of the study, the selection criteria for a more efficient combination of input data that can save time by omitting trial and error when building a data-driven model are presented.

연구논문 : 데이터마이닝 기법을 적용한 취수원 수질예측모형 평가

김주환 ( Ju Hwan Kim ),채수권 ( Soo Kwon Chae ),김병식 ( Byung Sik Kim ) 한국환경영향평가학회 2011 환경영향평가 Vol.20 No.5

For the efficient discovery of knowledge and information from the observed systems, data mining techniques can be an useful tool for the prediction of water quality at intake station in rivers. Deterioration of water quality can be caused at intake station in dry season due to insufficient flow. This demands additional outflow from dam since some extent of deterioration can be attenuated by dam reservoir operation to control outflow considering predicted water quality. A seasonal occurrence of high ammonia nitrogen (NH3-N) concentrations has hampered chemical treatment processes of a water plant in Geum river. Monthly flow allocation from upstream dam is important for downstream NH3-N control. In this study, prediction models of water quality based on multiple regression (MR), artificial neural network and data mining methods were developed to understand water quality variation and to support dam operations through providing predicted NH3-N concentrations at intake station. The models were calibrated with eight years of monthly data and verified with another two years of independent data. In those models, the NH3-N concentration for next time step is dependent on dam outflow, river water quality such as alkalinity, temperature, and NH3-N of previous time step. The model performances are compared and evaluated by error analysis and statistical characteristics like correlation and determination coefficients between the observed and the predicted water quality. It is expected that these data mining techniques can present more efficient data-driven tools in modelling stage and it is found that those models can be applied well to predict water quality in stream river systems.

Design and operation of a rainwater for drinking (RFD) project in a rural area: case study at Cukhe Elementary School, Vietnam

Dao, Anh Dzung,Nguyen, Duc Canh,Han, Moo Young IWA Publishing 2017 Journal of water, sanitation, and hygiene for deve Vol.7 No.4

<P>Since 2014, a well-designed rainwater for drinking (RFD) project has been successfully operating at Cukhe Elementary School, near Hanoi, Vietnam. During that time, daily rainfall data, water consumption, water quality and questionnaires to the community were prepared. Several concerns over the design and operation of RFD projects, such as lack of sufficient rainfall data, water quality concerns, and public acceptance, are identified and overcome. Modeled results from using observed daily rainfall data, and using a simplified method from insufficient monthly data, are compared. The simplified method using insufficient rainfall data is acceptable for design within the error range of 0–11%. Rainwater quality after the use of the point of use treatment device proved that a well-designed rainwater harvesting (RWH) system ensures safe drinking water, which complies with WHO and Vietnam drinking water quality standards (VDWQS) guidelines. The people of the community liked the RFD system because of the satisfactory water quality and the economic benefits of not needing to purchase bottled water. The success of the RFD project at the Cukhe Elementary School proved the potential of promoting rainwater as drinking water in rural areas in developing countries, where a safe drinking water supply is a challenge, and should be promoted as an important means to achieve Sustainable Development Goal 6.</P>

장기간(2005-2022) 자료 기반 대한민국 서울 지역 정수장 수질 변화 분석

오유나,손아정,김민정,정성필 적정기술학회 2024 적정기술학회지(Journal of Appropriate Technology) Vol.10 No.2

인류의 생활과 활동을 위해 전 세계적으로 안전하고 지속적인 물 공급이 필요하다. 기후변화와 인구증가 및 산업화 그리고 극한 가뭄과 홍수 같은 기후변화로 인하여, 물 부족 문제가 전 세계적으로 발생하고 있다. 이 문제는 특히 물 공급이 취약한 개발도상국이나 도서 지역에서 자주 발생하고 있다. 이 연구에서는 정수장이 안정적으로 운영되는지 평가하기 위하여, 2005년부터 2022년까지 서울시에서 운영되는 한 정수장에서 생산되는 수돗물, 유입수(한강), 직수 및 물탱크 저장수의 수질 측정 자료를 조사 및 활용하였다. 그 결과 한강 수계에서 붕소와 브롬 농도가 증가하는 것을 확인하였으며, 소독부산물 중 트리할로메탄계가 타 소독부산물(할로아세틱 에시드, 할로아세토니트릴, 할로알데하이드)에 비해 그 비율이 증가하는 것을 확인하였다. 소독부산물 중 브롬이 포함된 소독부산물이 증가하는 것으로 확인되었으나, 해당 기간동안 조사된 물의 경우 한국의 먹는물 수질 기준을 모두 만족함을 확인하였다. The safe and sustainable water supply is the one of the key aspects for human life and activity. Due to the extreme water variations such as drought and flooding by the climate change and increased water demand by the increased population and industry, the water scarcity became more severe globally. Therefore, it was required to understand the variations of the water qualities before and after the water production by using the long-term collected data. In this study, the measured and collected water quality data during 2005~2022 from the Seoul city, Korea for the feed water (Han river), produced water in water treatment plant (WTP), produced water at the end-point, and stored water at the end-point were investigated. The concentrations of boron and bromine in Han river were frequently and increasingly observed, respectively. In case of the disinfection byproducts (DBPs), the trihalomethanes (THMs) were increased while other species of the DBPs such as haloacetic acid (HAAs), haloacetonitriles (HANs), and haloaldehydes were decreased. The bromine included DBPs ratios also increased with time. All the water quality satisfied the water quality standard in Korea during 2005-2022.

패턴분류 방법 적용에 의한 장성호 수문,수질자료의 특성파악

박성천 ( Sung Chun Park ),진영훈 ( Young Hoon Jin ),노경범 ( Kyong Bum Roh ),김종오 ( Jong O Kim ),유호규 ( Ho Gyu Yu ) 한국물환경학회 2011 한국물환경학회지 Vol.27 No.6

Self Organizing Map (SOM) was applied for pattern classification of hydrological and water quality data measured at Jangseong Reservoir on a monthly basis. The primary objective of the present study is to understand better data characteristics and relationship between the data. For the purpose, two SOMs were configured by a methodologically systematic approach with appropriate methods for data transformation, determination of map size and side lengths of the map. The SOMs constructed at the respective measurement stations for water quality data (JSD1 and JSD2) commonly classified the respective datasets into five clusters by Davies-Bouldin Index (DBI). The trained SOMs were fine-tuned by Ward`s method of a hierarchical cluster analysis. On the one hand, the patterns with high values of standardized reference vectors for hydrological variables revealed the high possibility of eutrophication by TN or TP in the reservoir, in general. On the other hand, the clusters with low values of standardized reference vectors for hydrological variables showed the patterns with high COD concentration. In particular, Clsuter1 at JSD1 and Cluster5 at JSD2 represented the worst condition of water quality with high reference vectors for rainfall and storage in the reservoir. Consequently, SOM is applicable to identify the patterns of potential eutrophication in reservoirs according to the better understanding of data characteristics and their relationship.

자료기반 물환경 모델의 현황 및 발전 방향

차윤경 ( Yoonkyung Chaa ),신지훈 ( Jihoon Shinb ),김영우 ( Youngwoo Kimc ) 한국물환경학회 2020 한국물환경학회지 Vol.36 No.6

Although process-based models have been a preferred approach for modeling freshwater aquatic systems over extended time intervals, the increasing utility of data-driven models in a big data environment has made the data-driven models increasingly popular in recent decades. In this study, international peer-reviewed journals for the relevant fields were searched in the Web of Science Core Collection, and an extensive literature review, which included total 2,984 articles published during the last two decades (2000-2020), was performed. The review results indicated that the rate of increase in the number of published studies using data-driven models exceeded those using process-based models since 2010. The increase in the use of data-driven models was partly attributable to the increasing availability of data from new data sources, e.g., remotely sensed hyperspectral or multispectral data. Consistently throughout the past two decades, South Korea has been one of the top ten countries in which the greatest number of studies using the data-driven models were published. Among the major data-driven approaches, i.e., artificial neural network, decision tree, and Bayesian model, were illustrated with case studies. Based on the review, this study aimed to inform the current state of knowledge regarding the biogeochemical water quality and ecological models using data-driven approaches, and provide the remaining challenges and future prospects.

실시간 수치데이터의 이미지 기반 시각화 방식에 대한 연구 : 스마트폰 수질측정 어플리케이션을 대상으로

조은희(Cho Eunhee),김현욱(Kim Hyunook),류한영(Ryoo Han Young) 한국디지털디자인학회 2011 디지털디자인학연구 Vol.11 No.4

오늘날 4대강 사업 방사능 오염 등의 이슈로 인해 수질에 대한 현대인들의 관심이 급격하게 증가함에 따라 수질 속 오염 요소들의 수치를 실시간으로 모니터링 하는 작업이 보다 중요하게 여겨지고 있다. 현재 실시간 수질 데이터의 빠른 전송 및 정확한 분석을 위하여 스마트폰을 이용한 수질측정 어플리케이션의 개발이 시도되고 있으나 수치 값들을 단순히 텍스트로 나열해 놓는 1차원적인 정보제공 수준에 머물고 있어 스마트폰에서의 실시간 수치데이터의 시각화 기법에 대한 연구가 무엇보다 필요하다. 이에 본 연구에서는 수질측정 어플리케이션에 적용할 실시간 수치데이터의 이미지 기반 시각화 기법에 대해 살펴보고 이를 기반으로 하여 인터페이스 프로토타입을 제안하였다. Some issues such as 4-rivers restoration project radioactive contamination increase people's interests to the water quality. Therefore monitoring the degree of water's pollution factors in realtime has been an important issue. A smartphone application for measuring water quality has being developed to enable fast transmission and precise analysis of the realtime water quality data. However so far studies of visualizing the realtime data for smartphone have been very limited and the realtime data have been usually displayed as text-based interface on a smartphone. This study investigates the visualization methods for the realtime data which would be applied to the developing application. In addition based on the investigation results interface prototype is proposed.

Bayesian structural equation modeling for coastal management: The case of the Saemangeum coast of Korea for water quality improvements

Kim, Jinah,Park, Jinah Elsevier 2017 Ocean & coastal management Vol.136 No.-

<P><B>Abstract</B></P> <P>The data-driven paradigm is widely used in ocean science and includes the statistical modeling of various phenomena in coastal marine environments and data assimilation in numerical models. One of the most important challenges in the data-driven paradigm is finding the model that best approximates the underlying mechanism of a phenomenon with measurement data. In this paper, we propose a Bayesian approach to modeling coastal marine environments using ocean observational data, and we apply it to the Saemangeum coast. There are two main advantages to the Bayesian method: domain knowledge can be encoded to prior probability, and Markov-chain Monte Carlo simulation can be used in model estimation and inference. We apply the method to estimate model parameters and predict coastal water quality and sea current for maintaining optimal coastal water quality. The threshold quantity of sea current is computed to ensure sustainable coastal development. One of the interesting results we have obtained is a flat plateau relationship between the sea current for water exchange and the level of improvement of coastal water quality. This means that coastal water quality is not always being improved, even if the amount of water exchange is increased. The computed results are in good agreement with oceanographic theory, while showing a valid difference compared to the results using the frequentist approach and probabilistic inference using the probabilistic graphical model. These results will be helpful in coastal water quality management, ultimately contributing to sustainable coastal development.</P>