Design Characteristics of Studies Reporting the Performance of Artificial Intelligence Algorithms for Diagnostic Analysis of Medical Images: Results from Recently Published Papers

다국어 초록 (Multilingual Abstract)

Objective: To evaluate the design characteristics of studies that evaluated the performance of artificial intelligence (AI) algorithms for the diagnostic analysis of medical images.
Materials and Methods: PubMed MEDLINE and Embase databases were searched to identify original research articles published between January 1, 2018 and August 17, 2018 that investigated the performance of AI algorithms that analyze medical images to provide diagnostic decisions. Eligible articles were evaluated to determine 1) whether the study used external validation rather than internal validation, and in case of external validation, whether the data for validation were collected, 2) with diagnostic cohort design instead of diagnostic case-control design, 3) from multiple institutions, and 4) in a prospective manner. These are fundamental methodologic features recommended for clinical validation of AI performance in real-world practice. The studies that fulfilled the above criteria were identified. We classified the publishing journals into medical vs. non-medical journal groups. Then, the results were compared between medical and non-medical journalsResults: Of 516 eligible published studies, only 6% (31 studies) performed external validation. None of the 31 studies adopted all three design features: diagnostic cohort design, the inclusion of multiple institutions, and prospective data collection for external validation. No significant difference was found between medical and non-medical journals.
Conclusion: Nearly all of the studies published in the study period that evaluated the performance of AI algorithms for diagnostic analysis of medical images were designed as proof-of-concept technical feasibility studies and did not have the design features that are recommended for robust validation of the real-world clinical performance of AI algorithms.

참고문헌 (Reference)

1 박성호, "첨단 디지털 헬스케어 의료기기를 진료에 도입할 때 평가원칙" 대한의사협회 61 (61): 765-775, 2018

2 Greaves F, "What is an appropriate level of evidence for a digital health intervention?" 392 : 2665-2667, 2019

3 Zech JR, "Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a crosssectional study" 15 : e1002683-, 2018

4 Fryback DG, "The efficacy of diagnostic imaging" 11 : 88-94, 1991

5 Maddox TM, "Questions for artificial intelligence in health care" 321 : 31-32, 2019

6 Gianfrancesco MA, "Potential biases in machine learning algorithms using electronic health record data" 178 : 1544-1547, 2018

7 Park SH, "Methodologic guide for evaluating clinical performance and effect of artificial intelligence technology for medical diagnosis and prediction" 286 : 800-809, 2018

8 The Lancet, "Is digital medicine different?" 392 : 95-, 2018

9 Gill J, "Improving observational studies in the era of big data" 392 : 716-717, 2018

10 Luo W, "Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view" 18 : e323-, 2016

11 Korevaar DA, "Facilitating prospective registration of diagnostic accuracy studies: a STARD initiative" 63 : 1331-1341, 2017

12 Nsoesie EO, "Evaluating artificial intelligence applications in clinical settings" 1 : e182658-, 2018

13 Park SH, "Diagnostic case-control versus diagnostic cohort studies for clinical validation of artificial intelligence algorithm performance" 290 : 272-273, 2019

14 Li X, "Diagnosis of thyroid cancer using deep convolutional neural network models applied to sonographic images: a retrospective, multicohort, diagnostic study" 20 : 193-201, 2019

15 Ting DSW, "Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes" 318 : 2211-2223, 2017

16 Chartrand G, "Deep learning: a primer for radiologists" 37 : 2113-2131, 2017

17 AlBadawy EA, "Deep learning for segmentation of brain tumors: impact of cross-institutional training and testing" 45 : 1150-1158, 2018

18 이준구, "Deep Learning in Medical Imaging: General Overview" 대한영상의학회 18 (18): 570-584, 2017

19 Choy G, "Current applications and future impact of machine learning in radiology" 288 : 318-328, 2018

20 Yamashita R, "Convolutional neural networks: an overview and application in radiology" 9 : 611-629, 2018

21 박성호, "Connecting Technological Innovation in Artificial Intelligence to Real-world Medical Practice through Rigorous Clinical Validation: What Peer-reviewed Medical Journals Could Do" 대한의학회 33 (33): 1-7, 2018

22 INFANT Collaborative Group, "Computerised interpretation of fetal heart rate during labour (INFANT): a randomised controlled trial" 389 : 1719-1729, 2017

23 Shortliffe EH, "Clinical decision support in the era of artificial intelligence" 320 : 2199-2200, 2018

24 Rutjes AW, "Case-control and two-gate designs in diagnostic accuracy studies" 51 : 1335-1341, 2005

25 Tang A, "Canadian Association of Radiologists white paper on artificial intelligence in radiology" 69 : 120-135, 2018

26 SFR-IA Group, "Artificial intelligence and medical imaging 2018: French Radiology Community white paper" 99 : 727-742, 2018

27 Joseph R. England, "Artificial Intelligence for Medical Image Analysis: A Guide for Authors and Reviewers" American Roentgen Ray Society 212 (212): 513-519, 2019

28 강지훈, "Age of Data in Contemporary Research Articles Published in Representative General Radiology Journals" 대한영상의학회 19 (19): 1172-1178, 2018

29 "AI diagnostics need attention" 555 : 285-, 2018

30 Zou J, "AI can be sexist and racist - it’s time to make it fair" 559 : 324-326, 2018