Comparing the cut score for the borderline group method and borderline regression method with norm-referenced standard setting in an objective structured clinical examination in medical school in Korea

다국어 초록 (Multilingual Abstract)

Purpose: Setting standards is critical in health professions. However, appropriate standard setting methods do not always apply to theset cut score in performance assessment. The aim of this study was to compare the cut score when the standard setting is changed fromthe norm-referenced method to the borderline group method (BGM) and borderline regression method (BRM) in an objective structured clinical examination (OSCE) in medical school.
Methods: This was an explorative study to model the implementation of the BGM and BRM. A total of 107 fourth-year medical students attended the OSCE at 7 stations for encountering standardized patients (SPs) and at 1 station for performing skills on a manikinon July 15th, 2021. Thirty-two physician examiners evaluated the performance by completing a checklist and global rating scales.
Results: The cut score of the norm-referenced method was lower than that of the BGM (P<0.01) and BRM (P<0.02). There was nosignificant difference in the cut score between the BGM and BRM (P=0.40). The station with the highest standard deviation and thehighest proportion of the borderline group showed the largest cut score difference in standard setting methods.
Conclusion: Prefixed cut scores by the norm-referenced method without considering station contents or examinee performance canvary due to station difficulty and content, affecting the appropriateness of standard setting decisions. If there is an adequate consensuson the criteria for the borderline group, standard setting with the BRM could be applied as a practical and defensible method to determine the cut score for OSCE.

참고문헌 (Reference)

1 Charlotte Ringsted, "‘The research compass’: An introduction to research in medical education: AMEE Guide No. 56" Informa UK Limited 33 (33): 695-709, 2011

2 임미경, "Using the Angoff method to set a standard on mock exams for the Korean Nursing Licensing Examination" 한국보건의료인국가시험원 17 : 1-9, 2020

3 De Champlain AF, "Understanding medical education: evidence, theory, and practice" Wiley-Blackwell 347-359, 2018

4 Matthew Homer, "The impact of the inclusion of simulated patient ratings on the reliability of OSCE assessments under the borderline regression method" Informa UK Limited 31 (31): 420-425, 2009

5 Wood TJ, "Standard setting in a small scale OSCE: a comparison of the Modified Borderline-Group Method and the Borderline Regression Method" 11 : 115-122, 2006

6 박장희, "Similarity of the cut score in test sets with different item amounts using the modified Angoff, modified Ebel, and Hofstee standard-setting methods for the Korean Medical Licensing Examination" 한국보건의료인국가시험원 17 : 1-10, 2020

7 Matt Homer, "Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can ‘work’" Informa UK Limited 42 (42): 306-315, 2019

8 Min Liu, "Setting Pass Scores for Clinical Skills Assessment" Wiley 24 (24): 656-663, 2008

9 Danette W. McKinley, "How to set standards on performance-based examinations: AMEE Guide No. 85" Informa UK Limited 36 (36): 97-110, 2013

10 Vanda Yazbeck Karam, "Evaluating the validity evidence of an OSCE: results from a new medical school" Springer Science and Business Media LLC 18 (18): 2018

11 Rabab Abdel Ra’oof, "Developing a Model for Identification of the Borderline Group in Objective Structured Clinical Exam" Penerbit Universiti Sains Malaysia 12 (12): 1-10, 2020

12 M.T. Kane, "Designing and evaluating standard-setting procedures for licensure and certification tests" Springer Science and Business Media LLC 4 (4): 195-207, 1999

13 Janghee Park, "Comparison of standard-setting methods for the Korean Radiological Technologist Licensing Examination: Angoff, Ebel, bookmark, and Hofstee" Korea Health Personnel Licensing Examination Institute 15 : 32-, 2018

14 Reid KJ, "Comparing the borderline group and borderline regression approaches to setting objective structured clinical examination cut scores" 2 : 8-12, 2014

15 박송이, "Clinical performance of medical students in Korea in a whole-task emergency station in the objective structured clinical examination with a standardized patient complaining of palpitations" 한국보건의료인국가시험원 17 : 1-6, 2020

16 Norcini JJ, "Assessment methods in medical education" 23 : 239-250, 2007

17 Hejri SM, "Assessing the reliability of the borderline regression method as a standard setting procedure for objective structured clinical examination" 18 : 887-891, 2013

18 David M. Kaufman, "A Comparison of Standard-setting Procedures for an OSCE in Undergraduate Medical Education" Ovid Technologies (Wolters Kluwer Health) 75 (75): 267-271, 2000