논술고사의 신뢰성에 영향을 미치는 채점자 특성 분석 = Analyzing rater characteristics affecting the reliability of essay examinations

국문 초록 (Abstract)

  본 연구는 논술고사의 신뢰성에 채점자의 엄격성과 전문성이 미치는 영향을 알아보기 위해 다국면 라쉬 모형을 적용하여 대입 모의 논술고사를 분석하였다. 2회 실시된 동형의 논술고사에 참여한 피험자는 각각 647명과 873명이었으며, 채점은 동일한 10명의 교수에 의해 실시되었다. 그 중 4명은 전문가이고 나머지 6명은 비전문가에 속한다. 채점자 엄격성에 대한 분리신뢰도지수를 조사해본 결과는 채점자간에 유의미한 엄격성 차이가 존재한다는 것을 보여주었다. 그러나 전문가와 비전문가를 비교해볼 때, 전문가는 채점자간 엄격성 차이는 낮은 반면, 적합도는 높은 것을 알 수 있었다. 또한 전문가 집단은 2회에 걸친 논술고사에서 엄격성과 적합성이 비전문가보다 안정성있는 것으로 나타났다. 이는 논술고사의 신뢰성을 높이기 위해서는 채점자간 엄격성 차이를 고려해야 하며, 채점자의 전문성을 높이는 것이 선행되어야 한다는 것을 의미한다.

다국어 초록 (Multilingual Abstract)

  This paper analyzes the effects of rater "severity" and "expertise" on the reliability of essay examinations by using the many facet Rasch model(FACETS model). Two different essay examinations which administered to different student groups at a university were used for this study. The two examinations included three tasks having the same format and the same scoring method. There were 647 examinees in the first essay examination and 873 in the second one. Ten raters participated in both the first and the second examination. Among them, four raters could be considered as experts and six raters as non-experts in terms of their major content and rater experience. The data were analyzed by FACETS computer program with three facets of examinee, task, and rater. The separation reliability statistics and chi-square tests produced by FACETS showed that there were significant differences between raters for severity. From the comparison between experts and non-experts, it was found that experts tended to have better fits than non-experts. This means that experts can have higher intra-rater reliability. In addition, it was also disclosed that experts were more stable for severity and infit over two administrations. All these results suggest that the factors of rater severity and rater expertise should be seriously considered for the reliability of essay scoring. Especially, it should be recognized that rater severity can be influenced by rater expertise. This implies that rater trainings are required to improve the reliability of essay scoring.

목차 (Table of Contents)

• Ⅰ. 서론
  Ⅱ. 이론적 배경
  Ⅲ. 연구방법
  Ⅳ. 연구결과
  Ⅴ. 결론 및 논의
  참고문헌
  저자소개
  〈ABSTRACT〉
  

참고문헌 (Reference)

1 최연희, "채점자 훈련이 영어 작문 채점에 미치는 효과에 대한 연구: FACETS 분석을 통한 신뢰도 변화 분석" 18 (18): 257-291, 2002

2 채선희, "문항반응이론의 이론과 실제" 서현사 2003

3 박정, "대규모 평가 자유반응형 문항 채점 영향 요소" 18 (18): 35-57, 2005

4 성태제, "논술형 고사와 예체능계 실기고사를 위한 채점자간 신뢰도 추정" 7 (7): 43-56, 1994

5 채선희, "논술시험 채점의 공정성과 효율성 확보방안-채점과정의 엄격성과 FACETS 모형에 의한 채점결과 분석" 9 (9): 5-29, 1996

6 김성숙, "논술문항 채점의 변동요인 분석과 일반화가능도 계수의 최적화 조건" 8 (8): 35-57, 1995

7 지은림, "many-facet Rasch 모형을 적용한 대입 논술고사 채점의 객관성 연구" 9 (9): 5-22, 1996

8 Lynch, B. K., "Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants" 15 : 158-180, 1998

9 Weigle,S.C., "Using FACETS to Model Rater Trainging Effects" 1994

10 Brown,A., "T도 effects of rater variables in the development of an occupation specific language performance test" 12 (12): 1-15, 1995

11 Congdon, P. J., "The stability of rater severity in large-scale assessment programs" 37 (37): 163-178, 2000

12 Engelhard,G., "The measurement of writing ability with a many-facet Rasch model" 1991

13 Lunz, M. E., "The invariance of rater severity calibrations in:Objective Measurement: Theory into practice" Ablex Publishing 99-112, 1996

14 Liu, K., "Rater effects in the consensual assessment of creative products" 2008

15 Lumley, T., "Rater characteristics and rater bias: implications for training" 12 : 54-71, 1995

16 지은림, "Rasch 모형의 이론과 실제.교육,심리,사회,연구법 총서" 교육과학사 2001

17 Lunz, M. E., "Measuring the impact of judge severity on examination scores" 3 (3): 331-345, 1990

18 Linacre,J.M., "Many-facet Rasch measurement" MESA Press 1988

19 Stahl, J. A., "Judge performance reports: Media and message" 1991

20 Linacre,J.M., "FACETS: Computer Program for Analysis of Examinations with Multiple FACETS" MESA Press 2003

21 Wigglesworth, "Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction" 10 : 305-335, 1993

22 Engelhard,G., "Evaluating rater accuracy in performance assessments" 33 (33): 56-70, 1996

23 Myford, C., "Detecting and measuring rater effects using many-facet Rasch measurement" 5 (5): 189-227, 2004

24 Andrich,D., "A rating formulation for ordered response categories" 43 : 561-573, 1978

25 DeCarlo,L.T., "A model of rater behavior in essay grading based on signal detection theory" 42 (42): 53-76, 2005

26 Kimi,Kondo-Brown, "A FACET analysis of rater bias in measuring Japanese second language writing performance" 19 (19): 1-31, 2002