Receiver operating characteristic curve: overview and practical use for clinicians

Nahm Francis Sahngun 대한마취통증의학회 2022 Korean Journal of Anesthesiology Vol.75 No.1

Using diagnostic testing to determine the presence or absence of a disease is essential in clinical practice. In many cases, test results are obtained as continuous values and require a process of conversion and interpretation and into a dichotomous form to determine the presence of a disease. The primary method used for this process is the receiver operating characteristic (ROC) curve. The ROC curve is used to assess the overall diagnostic performance of a test and to compare the performance of two or more diagnostic tests. It is also used to select an optimal cut-off value for determining the presence or absence of a disease. Although clinicians who do not have expertise in statistics do not need to understand both the complex mathematical equation and the analytic process of ROC curves, understanding the core concepts of the ROC curve analysis is a prerequisite for the proper use and interpretation of the ROC curve. This review describes the basic concepts for the correct use and interpretation of the ROC curve, including parametric/nonparametric ROC curves, the meaning of the area under the ROC curve (AUC), the partial AUC, methods for selecting the best cut-off value, and the statistical software to use for ROC curve analyses.

ROC곡선을 이용한 중년여성의 건강체력평가

이창진(Chang Jin Lee) 한국발육발달학회 2016 한국발육발달학회지 Vol.24 No.1

Receiver operating characteristic(ROC) curve, an index that is actively used since the 1960s, and is used to verify the classification accuracy. The aim of present study was to establish the levels of healthrelated physical fitness using a ROC in middle age women. Participants were categorized into either a exercise group (n=80) or a non-exercise group(n=70). In order to examine the physical fitness level of the participants, in the present study was tested physical fitness factors including muscle strength, back muscle strength, muscular endurance, flexibility, cardiopulmonary endurance and body composition. The ROC curve was analyzed using SPSS 20.0. The area under the ROC curve (AUC) of physical fitness factors were muscle strength .501, back muscle strength .489, muscular endurance .634, flexibility .560, cardiopulmonary endurance .425 and body composition .505. Moreover, the sensitivity and specificity of muscle strength in 27.2 kg cutoff value was 51.1% and 45.6%, respectively. In the back muscle strength in 66.2 kg cutoff value, sensitivity and specificity was calculated by 48.9% and 53.3%, respectively. The sensitivity and specificity for muscular endurance in 9.5 cutoff value repetitions was 66.7% and 48.9%, respectively. The sensitivity and specificity for cardiopulmonary endurance in 67.6 second cutoff value was observed by 48.8% and 57.7%, respectively. Finally, cutoff value of percentage body fat, as a body composition was 29.9 %, and sensitivity and specificity was calculated by 51.5% and 52.6%, respectively.

성격점수를 이용한 ROC-curve 기반 사상체질 분류 방법에 대한 연구

김호석(Hoseok Kim),장은수(Eunsu Jang),김상혁(Sanghyuk Kim),유종향(Jonghyang Yoo),이시우(Siwoo Lee) 한국한의학연구원 2011 한국한의학연구원논문집 Vol.17 No.2

Objectives : Sasang typology is extensively studied for the Sasang constitution diagnosis objectification with various data, for example, questionaires, reference materials , etc and analyzed with the several statistical methods. In this study, we used ROC-curve (Receiver Operating Characteristic curve) analysis to diagnose Sasang constitution, which is a kind of epidemiologic research methods and is away from traditional statistical methods. Methods : We collected personality questionnaire which consists of 15 items, from 24 oriental medical clinics. We analyzed the sensitivity and specificity using ROC curve method based on the score of personality questionnaire and also investigated classification accuracy and cut-off value of Sasang constitution. Results : The AUC (area under the ROC curve) value was 0.508 (p=.5511) for Taeeumin, 0.629 (p<.0001) for Soeumin and 0.604(p<.0001) for Soyangin, respectively. so the classification accuracy for Soeumin was highest Soeumin for over 30 points and Soyangin for below 28 points respectively. Conclusions : We suggest that Taeeumin is not classified easily in the ROC-curve analysis. We may classify Soeumin and Soyangin but the accuracy of Sasang constitutional diagnosis is still low

디지털 흉부 방사선 영상에서 Hybrid Filter와 Inverse Filter를 적용한 종양의 검출능 평가

김윤영(Youn-Young Kim),김태영(Tae-Young Kim),김현지(Hyun-Ji Kim),박민석(Min-Seock Park),김정민(Jung-Min Kim) 대한방사선과학회(구 대한방사선기술학회) 2013 방사선기술과학 Vol.36 No.4

본 연구에서는, 흉부단순사진에 있어서 종양그림자의 검출에 대한 흑백 Inverse image과 Hybrid image의 유용성을 ROC해석으로 검토하였다. 증례의 선택은 일본방사선기술학회가 발행한 표준 Digital 영상 Date Base로부터 30장을 선택 하여 original image로 하였다. c언어를 통해 Inverse image는 60장, Hybrid image는 30장 제작하였다. 실험방법으로 연속 판독실험을 하였고, ROC실험 display program은 Matlap을 통하여 작성하였다. 관찰자의 수는 Inverse image의 경우 방사선사 5명과 방사선의 2명, 합계 7명으로 실험하였다. Hybrid 영상의 경우, 방사선 전공자 3명과 숙련된 방사선사 2명, 합계 5명으로 실험하였다. ROC곡선은 Metz가 작성한 ROCKIT Program을 이용하여 구하였다. Inverse image의 경우 관찰자 7명 전원, 방사선과의 2명, 방사선사 5명의 평균 ROC곡선의 Az는 각각 original image의 0.742, 0.793, 0.721에서, Inverse image의 0.775, 0.821, 0.753까지로, 통계적 유의차로 증가하였다. Hybrid image의 경우 관찰자 5명 전원, 숙련된 방사선사 2명, 방사선학 전공자 3명의 평균 ROC곡선의 Az는 각각 original image의 0.525, 0.491, 0.5478에서, Hybrid 영상의 0.4868, 0.539, 0.450로 변화 하였다. 결론적으로, 흉부단순사진에서 종양의 검출에 관하여, Inverse image은 유의하지만, Hybrid 영상의 경우 유의한 차이가 나타나지 않았다. The purpose of this study is to evaluate usefulness of Hybrid image and Inverse image about detection of tumor shadow in chest radiography using ROC analysis. Original images of 60 cases are selected from Standards digital image date base issued by the Japanese Society of Radiological Technology. Through computer language of C, Inverse images of 60 cases and Hybrid image of 30 cases are made. The continues reading experiment was conducted. In the case of inverse image were observed by 5 radiographer and 2 radiologist. In the case of In case of Hybrid image were observed by 3 student radiographer and 2 experienced radiographer. ROC curve are constructed using ROCKIT Program made by Metz. In Inverse image, a Az of average ROC curve was increases from 0.742 of original image to 0.775 of inverse image. In normal cases, the effect of the detrimental is same to that of the beneficial, however In abnormal cases, the beneficial effect is greater than detrimental effect. However in Hybrid image, a Az of average ROC curve was decreases from 0.5253 of original image to 0.4868 of Hybrid image. In Normal cases, the effect of the detrimental is greater than that of the Beneficial, however In abnormal cases, the Beneficial effect is greater than detrimental effect. The inverse image can be more positively considered for the detecting of tumor than the hybrid image.

동두천 거주 초등학생의 시각 선별 검사를 위한 자동굴절검사의 타당도 분석

황혜경,이규병,박지현,마기중 대한시과학회 2011 대한시과학회지 Vol.13 No.1

목 적: 본 연구는 ROC curve를 분석하여 나안시력과 자동굴절검사의 민감도/특이도에 따른 최상의 절단점(Optimal cutoff point)을 산출하고, 자동굴절검사가 선별검사(Screening)로 타당한지 알아보고자 한다. 방 법: 초등학생 293명을 대상으로 나안시력, 자동굴절검사, 자각적 굴절검사를 실시하여, 나안시력 0.5를 기준으로 자각적 굴절검사에 대한 최상의 절단점을 산출하고 이를 황금기준(GoldStandard)으로 가정한 후 이 절단점 및 인접한 값을 기준으로 정상/추가검사의뢰를 분류하였고, 나안시력과 자동굴절검사 결과에 대해 ROC curve를 이용하여 민감도/특이도가 가장 높은 최상의 절단점을 산출하였다. 또한 곡선아래면적(Area Under the Curve)을 이용하여 자각적 굴절검사의 분류 결과와 차이가 있는지 확인하여 자동굴절검사가 선별검사로서 타당한지 알아보았다. 결 과: ROC Curve를 이용하여 나안시력 0.5를 기준으로 한 자각적 굴절검사의 최상의 절단점은 -0.63D로 나타났다. 자각적 굴절검사 -0.75D를 황금기준으로 가정할 때 절단점은 나안시력은 0.5, 자동굴절검사는 -1.00D로 나타났으며, 두 검사법을 비교한 결과 나안시력과 자동굴절검사의 곡선아래면적은 통계적으로 차이가 없었다. 결 론: 어린이의 시각 선별검사를 위해 나안시력과 함께 자동굴절검사를 활용할 수 있을 것으로 생각된다. Purpose: To assess the optimal cutoff point of visual acuity, auto refraction which is both high sensitivity and specificity with ROC curve analysis, and to evaluate the validity of auto refraction for vision screening of elementary school children. Methods: 553 eyes of elementary students were recruited and tested visual acuity, auto refraction and subjective refraction. The subjective refraction was considered as gold standard to differentiate normal or referral, and the sensitivity/specificity and cutoff point of the tests was calculated with ROC analysis. The accuracy of tests were assessed with comparing Area under the curve(AUC) of ROC curve and evaluated the validity of auto refraction whether it is useful for vision screening of elementary school children. Results: The optimal cutoff point of subjective refraction for decimal visual acuity 0.5 criteria is -0.63D. Therefore, gold standard was assumed to be -0.75D respectively. For subjective refraction -0.75D criteria, optimal cutoff point of visual acuity was 0.5, auto refraction was -1.00D, respectively. The AUC of the tests were not significantly different(p = 0.118). For subjective refraction -0.50D criteria, optimal cutoff point of visual acuity was 0.7, auto refraction was -0.78D, respectively. The AUC of visual acuity and auto refraction were not significantly different(p= 0.923). Conclusion: When subjective refraction was assumed as gold standard and compared AUC of visual acuity with auto refraction, the accuracy of tests were not significantly different. therefore, auto refraction and unaided visual acuity were both considered as useful tools of vision screening.

The Analysis of Convergence Insufficiency using the CISS Questionnaire in Those Non-strabismic Horizontal Misalignment

Jin Yeong Kim,Koon-Ja Lee 대한시과학회 2018 대한시과학회지 Vol.20 No.1

목적: 비사시성 수평 안위 이상안을 대상으로 원, 근거리 사위군에서 폭주부족을 CISS(Convergence insufficiency symptom survey) 설문지를 이용하여 측정하고 상관성을 분석하고자 하였다. 방법: 경남 김해에 거주하고 굴절이상이 구면도수 ±6.00 D 이하, 원주도수 ±2.00D 이하인 대학생 94명 (남; 36명, 여; 58명, 평균나이 22.56 ± 3.72세)을 대상으로 하였으며, 전신질환 및 안질환이 있거나, 양안의 차이가 2.00 D 이상인 굴절이상안은 제외하였다. 비사시성 수평안위이상 검사를 위해 완전융합제거사위는 마 독스 로드를 이용하였고, 일부융합제거사위는 Wesson 주시시차 카드를 이용하였으며, CISS 설문결과에 따라 폭주부족의 증상군과 비증상군으로 나누었다. 통계분석은 SPSS 22.0을 이용하였고, CISS 증상군과 비증상군의 사위도에 따른 평균 값 비교는 Mann-Whitney U test를 이용하였으며 사위의 분류에 따른 비율은 교차 분석을 이용하였으며, ROC curve는 통계프로그램 Medcalc version 17.5.5(Medcalc Software, Mariakerke, Belgium)를 사용하여 민감도, 특이도 및 AUC 값을 구하여 비교하였고 유의수준은 p<0.05로 하였다. 결과: 원거리 완전융합제거사위 검사에서 외사위군(n=62)에서는 CISS 증상군이 36명(43.4%), 비증상군 이 26명(31.3%)이며, 내사위군(n=21)에서는 증상군이 12명(14.5%), 비증상군이 9명(10.8%)으로 나타났으며, 원거리 사위 종류에 따라서 CISS 폭주부족 증상군과 비증상군의 비율은 차이가 없었다(χ2 =0.131, p=0.937). 근거리 완전융합제거사위 검사에서 외사위군(n=64)에서 증상군 37명(39.4%), 비증상군 27명(28.7%), 내사위 군(n=18)에서 증상군은 12명(12.8%), 비증상군은 6명(6.4%)으로 근거리 사위종류에 따라 증상군과 비증상 군의 비율은 차이가 없었다(χ2 =0.424, p=0.809). 일부융합제거사위 검사에서 외편위군(n=61)에서는 증상군 이 36명(38.3%), 비증상군은 25명(26.6%), 내편위군(n=10)에서 증상군은 6명(10.7%), 비증상군은 4명(10.5%) 으로 나타났으며, 편위의 종류에 따라 증상군과 비증상군의 비율은 차이가 없었다(χ2 =2.474, p=0.290). 사위검사방법의 민감도와 특이도 및 AUC는 각각 원거리 완전융합제거검사 41.3%, 75.2%, 0.542, 근거리 완전융합제거검사 69.7%, 53.1%, 0.560, 일부융합제거검사 87.2%, 특이도 2.4%, 0.541로 나타났지만 검사법에 따라 차이가 없었다. 결론: 원, 근거리 완전융합제거사위 검사로 분류한 사위의 종류에 따라 CISS 설문지로 조사한 폭주부족 증상군과 비증상군의 비율은 차이가 없었고, 일부융합제거 사위 검사로 분류한 편위의 종류에 따라서도 차이가 없었다. Subjects: To analyze the correlation between the degree of distance and near phoria and convergence insufficiency using the CISS (convergence insufficiency symptom survey) questionnaire in those with non-strabismic horizontal misalignment. Methods: A total of 94 college students (36 males and 58 females, mean age: 22.56 ± 3.72 years) in Gimhae, Gyeongnam province were enrolled. The refractive error of the subject was spherical power below ± 6.00 D and cylinder power below ± 2.00 D. Those with systemic and ocular diseases and more than 2 D difference between the two eyes were excluded from this study. Associated phoria and dissociated phoria tests were performed using the Maddox rod and the Wesson fixation disparity card, respectively. According to the results of the CISS questionnaire, the subjects were divided into two groups as being symptomatic and asymptomatic. Statistical analysis was performed using SPSS 22.0, A Mann-Whitney U test was applied to compare the mean values according to the degree of phoria between CISS symptomatic and asymptomatic group. The sensitivity, specificity and AUC (area under an ROC curve) of the phoria tests are compared using statistical program Medcalc version 17.5.5 (Medcalc Software, Mariakerke), and the significance level was p<0.05. Results: In the distance exophoria group (n=62), the number of CI (convergence insufficiency) symptomatic group was 36 (43.4%) and CI asymptomatic group was 26 (31.3%) and in the distance esophoria group (n=21) CI symptomatic group was 12 (14.5%), and CI asymptomatic group was 9 (10.8%). There was no significant difference between CI symptomatic and asymptomatic group in the associated phoria group at distance (χ2 =0.131, p=0.937). In the near exophoria group (n=64) CI symptomatic and asymptomatic groups were 37 (39.4%) and 27 (28.7%), respectively, and in the near esophoria group (n=18), CI symptomatic and asymptomatic groups were 12 (12.8%) 6 (6.4%), respectively. There was no significant difference between CI symptomatic and CI asymptomatic groups according to the associated phoria at near (χ2 =0.424, p=0.809). In exo-deviation group(n=61), thirty-six (38.3%) were CI symptomatic and twenty-five (26.6%) were CI asymptomatic, p and in the eso-deviation group (n=10), six (10.7%) were CI symptomatic and four (10.5%) were CI asymptomatic. There was no significant difference between CI symptomatic and asymptomatic groups in the dissociated phoria test (χ2 =2.474, p=0.290). In the associated phoria test, the ROC curve showed a sensitivity, specificity and AUC of 41.3% and 75.2%, 0.524 at distance and 69.7%, 53.1% and 0.560 at near, respectively. In the dissociated phoria test, the ROC curve showed a sensitivity of 87.2% and a specificity of 2.4% and AUC of 0.541, showing that there was no significant difference among phoria tests. Conclusions: There was no significant difference in the ratio of the CI symptomatic and asymptomatic groups using the CISS questionnaire according to the associated phoria tests at distance and near. There was no correlation between the fixation disparity and CI symptomatic and asymptomatic groups.

Parametric ROC Analysis to Evaluate Image Reconstruction Algorithms for Computed Tomography

Rhee, Seong-Won,Park, Joon-ho,Oh, Ju-Hwan 경운대학교 산업기술연구소 2001 産業技術硏究論文誌 Vol.3 No.2(C)

When a diagnostic test is based on an observed variable that lies on a continuous or graded scale, and assessment of the overall value of the test can be made through the use of a receiver operating characteristic (ROC) curve. Parametric estimator assuming Gaussian distribution accesses to the true area under ROC curve (AUC), while nonparametric estimator has a systematical underestimation problem. In this paper, we consider a parametric approach to analyze areas under correlated ROC curves and compare this parametric method with Hanley and McNeil(1983)approach and DeLong et al.(1988) approach.

ROC(receiver operating characteristics) 해석

김재덕,Kim Jae-Duk 대한영상치의학회 2000 Imaging Science in Dentistry Vol.30 No.3

1. 일반방사선사진과 칼라화한 방사선사진의 비교에서 각각 필름에서 진단을 시행할 때 ROC해석법에서는 true positive fraction (TPF), false positive fraction (FPF)를 매개변수로 하고 있으므로 우선 두가지 필름형태에 대해 각각 따로 다음과 같이 평가한다. 2. 판정기준 병변없다 A, 거의 없다 B, 모르겠다 C, 거의 있다 D, 있다 E 먼저 일반방사선사진에서 실제로 병소가 총있는 것이 50, 총없는 것이 50인데 위 판정기준 각각에 대해(equation omitted) 3. 곡선만들기 a.횡축은 FPF 종축은 TPF로 한 그래프를 plot를 한다. sensitivity 17/50 specificity 26/50 accuracy 43/100 b. 곡선만들기 프로그램을 이용하여 곡선을 만들시에는 TPF를 a에 입력하고 PFP를 b에 입력한다. 이 plot을 그릴 수 있는 프로그램은 http://www.members.tripod.co.kr/jdakim 또는 http://www.chosun.ac.kr/∼jdakim의 홈페이지내 공개자료실에서 다운 받으실 수 있습니다. (equation omitted) 이 프로그램에서 입력할 a, b의 값은 (equation omitted) 위와같이 입력하여 얻어진 일반방사선사진에서의 판독 결과 얻어진 곡선이 그래프에서 곡선이 된다. 이와 같은 커브를 컬러화한 사진 판독에서 똑같이 시행하여 ROC곡선(윗곡선)을 만든 다음 두 곡선을 비교하여 아래면적이 더 큰 쪽이 병소 판독에 우수하다고 결론짓는다. The purpose of this paper is to explain the making procedure and the usage of receiver operating characteristic (ROC) curve for interpretation of radiographic images. The conventional radiograms obtained after the creation of the lesions in the acrylic plates and were enhanced in color. The observer were informed of which tooth to examine, the 'a priori' probability of a lesion present and the approximate diameter of the lesions. The two groups of films were interpreted separately by the same observer using the same rating scale. The following rating scale was used: A; definitely no lesion, B; probably no lesion, C; not sure, D; probably a lesion, and E; definitely a lesion. In analysis, for each observer the diagnostic results in terms of true positive (TP) and false positive (FP) decisions were plotted on a graph. The lowest point on the graph represents the TP and FP when only decisions designated as E according to the rating scale are included. The next point shows the TP and FP values when diagnoses designated as D are added and so forth. By connecting such plot points, a receiver operating characteristic (ROC) curves is obtained. The area under the curve represents the diagnostic accuracy resulting from a diagnostic performance at pure chance level and a value of 1.0 at perfect performance. This method has been known as an useful method to detect the minute difference for each radiographic technic, each observer and for the different lesion depths.

Estimating the AUC of the MROC curve in the presence of measurement errors

Siva G,Vishnu Vardhan R 한국통계학회 2022 Communications for statistical applications and me Vol.29 No.5

Collection of data on several variables, especially in the field of medicine, results in the problem of measurement errors. The presence of such measurement errors may influence the outcomes or estimates of the parameter in the model. In classification scenario, the presence of measurement errors will affect the intrinsic cum summary measures of Receiver Operating Characteristic (ROC) curve. In the context of ROC curve, only a few researchers have attempted to study the problem of measurement errors in estimating the area under their respective ROC curves in the framework of univariate setup. In this paper, we work on the estimation of area under the multivariate ROC curve in the presence of measurement errors. The proposed work is supported with a real dataset and simulation studies. Results show that the proposed bias-corrected estimator helps in correcting the AUC with minimum bias and minimum mean square error.

수위관측소의 유지관리를 위한 로지스틱 회귀모형 구축 및 평가

이지호(Lee Ji Ho),이정민(Lee Jung Min),천정환(Chun Jung Hwan),구본상(Koo Bon Sang),전환돈(Jun Hwan Don) 한국방재학회 2017 한국방재학회논문집 Vol.17 No.2

본 연구에서는 효율적인 수위관측소의 유지관리를 위해 수위관측소의 문제점을 예측할 수 있는 로지스틱 회귀모형을 제시하고, 산정된 회귀모형의 효용성을 ROC 성능평가기법을 통해 평가하였다. 제안된 방법론을 한강유역에 대해 적용하여 그 효율성을 검토하였다. 그 결과를 정리하면 다음과 같다. 회귀모형 구축을 위해 2013-2015년의 수위유지관리 자료를 수집하였으며, 독립변수와 종속변수와의 관계를 고려하여 6개의 독립변수를 선별하였다. 2013-2014년의 자료로 구축된 회귀모형을 2015년의 자료예측에 적용한 결과 83.97%의 예측률을 보였다. 구축된 회귀모형을 ROC 성능평가기법을 통해 모형의 우수함을 확인하였다. 아울러 본 연구에서 제안된 방법론은 관리가 필요한 수문관측지점을 선별할 수 있는 의사결정 기법으로 활용될 수 있다. In this study, we propose a logistic regression model to predict the problems of the water level stations for the efficient maintenance of water level stations, then the effectiveness of the estimated logistic regression model is evaluated by ROC curve. The results are summarized as follows. In order to construct a regression model, water level maintenance data for 2013-2015 is collected and 6 independent variables are selected with considering the relationship between independent variables and dependent variables. As a result of applying regression model constructed with the data of 2013-2014, the prediction rate for data prediction of 2015 year is evaluated by 83.97%. It is found that the constructed regression model is confirmed to be suitable using the ROC curve analysis. The methodology proposed in this study can be used as a decision making technique to distinguish hydrologic observation points that need to be managed.