A Novel Fundus Image Reading Tool for Efficient Generation of a Multi-dimensional Categorical Image Database for Machine Learning Algorithm Training

다국어 초록 (Multilingual Abstract)

Background: We described a novel multi-step retinal fundus image reading system for providing high-quality large data for machine learning algorithms, and assessed the grader variability in the large-scale dataset generated with this system.
Methods: A 5-step retinal fundus image reading tool was developed that rates image quality, presence of abnormality, findings with location information, diagnoses, and clinical significance. Each image was evaluated by 3 different graders. Agreements among graders for each decision were evaluated.
Results: The 234,242 readings of 79,458 images were collected from 55 licensed ophthalmologists during 6 months. The 34,364 images were graded as abnormal by at-least one rater. Of these, all three raters agreed in 46.6% in abnormality, while 69.9% of the images were rated as abnormal by two or more raters. Agreement rate of at-least two raters on a certain finding was 26.7%–65.2%, and complete agreement rate of all-three raters was 5.7%–43.3%. As for diagnoses, agreement of at-least two raters was 35.6%–65.6%, and complete agreement rate was 11.0%–40.0%. Agreement of findings and diagnoses were higher when restricted to images with prior complete agreement on abnormality.
Retinal/glaucoma specialists showed higher agreements on findings and diagnoses of their corresponding subspecialties.
Conclusion: This novel reading tool for retinal fundus images generated a large-scale dataset with high level of information, which can be utilized in future development of machine learning-based algorithms for automated identification of abnormal conditions and clinical decision supporting system. These results emphasize the importance of addressing grader variability in algorithm developments.

참고문헌 (Reference)

1 Quellec G, "Optimal wavelet transform for the detection of microaneurysms in retina photographs" 27 (27): 1230-1241, 2008

2 Arunkumar R, "Multi-retinal disease classification by reduced deep learning features" 28 (28): 329-334, 2017

3 Choi JY, "Multi-categorical deep learning neural network to classify retinal images: a pilot study employing small database" 12 (12): e0187336-, 2017

4 Abràmoff MD, "Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning" 57 (57): 5200-5206, 2016

5 Krause J, "Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy" 125 (125): 1264-1272, 2018

6 Decencière E, "Feedback on a publicly distributed image database: the Messidor database" 33 (33): 231-234, 2014

7 Ting DS, "Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes" 318 (318): 2211-2223, 2017

8 Gulshan V, "Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs" 316 (316): 2402-2410, 2016

9 Quellec G, "Deep image mining for diabetic retinopathy screening" 39 : 178-193, 2017

10 박성호, "Connecting Technological Innovation in Artificial Intelligence to Real-world Medical Practice through Rigorous Clinical Validation: What Peer-reviewed Medical Journals Could Do" 대한의학회 33 (33): 1-7, 2018

11 Gargeya R, "Automated identification of diabetic retinopathy using deep learning" 124 (124): 7-962, 2017

12 Wong TY, "Artificial intelligence with deep learning technology looks into diabetic retinopathy screening" 316 (316): 2366-2367, 2016

13 Takahashi H, "Applying artificial intelligence to disease staging: deep learning for improved staging of diabetic retinopathy" 12 (12): e0179790-, 2017