Acceptability of the 8-case objective structured clinical examination of medical students in Korea using generalizability theory: a reliability study

다국어 초록 (Multilingual Abstract)

Purpose This study investigated whether the reliability was acceptable when the number of cases in the objective structured clinical examination (OSCE) decreased from 12 to 8 using generalizability theory (GT).
Methods This psychometric study analyzed the OSCE data of 439 fourth-year medical students conducted in the Busan and Gyeongnam areas of South Korea from July 12 to 15, 2021. The generalizability study (G-study) considered 3 facets—students (p), cases (c), and items (i)—and designed the analysis as p×(i:c) due to items being nested in a case. The acceptable generalizability (G) coefficient was set to 0.70. The G-study and decision study (D-study) were performed using G String IV ver. 6.3.8 (Papawork, Hamilton, ON, Canada).
Results All G coefficients except for July 14 (0.69) were above 0.70. The major sources of variance components (VCs) were items nested in cases (i:c), from 51.34% to 57.70%, and residual error (pi:c), from 39.55% to 43.26%. The proportion of VCs in cases was negligible, ranging from 0% to 2.03%.
Conclusion The case numbers decreased in the 2021 Busan and Gyeongnam OSCE. However, the reliability was acceptable. In the D-study, reliability was maintained at 0.70 or higher if there were more than 21 items/case in 8 cases and more than 18 items/case in 9 cases. However, according to the G-study, increasing the number of items nested in cases rather than the number of cases could further improve reliability. The consortium needs to maintain a case bank with various items to implement a reliable blueprinting combination for the OSCE.

참고문헌 (Reference)

1 Schuwirth LW, "Understanding medical education: evidence, theory, and practice" Wiley-Blackwell 275-289, 2018

2 Schuwirth LW, "The use of clinical simulations in assessment" (37 Suppl 1) : 65-71, 2003

3 Newble D, "Techniques for measuring clinical competence : objective structured clinical examinations" 38 : 199-203, 2004

4 Vaughan B, "Reliability of a viva assessment of clinical reasoning in an Australian pre-professional osteopathy program assessed using generalizability theory" 14 : 1-, 2017

5 Downing SM, "Reliability : on the reproducibility of assessment data" 38 : 1006-1012, 2004

6 Blood AD, "Neurology objective structured clinical examination reliability using generalizability theory" 85 : 1623-1629, 2015

7 Norman G, "How specific is case specificity" 40 : 618-623, 2006

8 Brennan RL, "Generalizability theory" 11 : 27-34, 1992

9 Crossley J, "Generalisability : a key to unlock professional assessment" 36 : 972-978, 2002

10 Swanson DB, "Factors influencing reproducibility of tests using standardized patients" 1 : 158-166, 1989

11 Kreiter CD, "Case specificity : empirical phenomenon or measurement artifact" 19 : 378-381, 2007

12 Brannick MT, "A systematic review of the reliability of objective structured clinical examination scores" 45 : 1181-1189, 2011