The Effects of a Rubric on Inexperienced Raters’ Scoring Consistency

이경랑 현대영어교육학회 2016 현대영어교육 Vol.17 No.2

This study aimed to explore whether a rubric was effective for promoting raters’ scoring consistency when evaluating English learners’ essays. The experienced raters’ scores of Korean university students, with a rubric, were analyzed qualitatively and quantitatively. The scores of inexperienced raters with the rubric were compared with those without the rubric. Each group’s scores were compared using t-tests (also with inter-rater reliability tests) and the inexperienced groups’ scores were compared using ANOVA. With the rubric, the inexperienced raters showed a similar way of evaluating students’ performance to the experienced raters while those without the rubric could not. The former scored the essays consistently both within and across the semesters, but the latter could do so only within the semesters (post-test scores were higher than pre-test scores within each semester). The scores of each test given by the inexperienced raters with the rubric were not statistically different, which was not true for the experienced with the rubric. This implies that rubrics are effective in promoting inexperienced raters’ scoring consistency, especially when raters evaluate essays at the same time rather than when evaluating at different times even with a rubric. The details of the results are given, and the significance is discussed.

담화 능력 평가를 위한 채점기준표 개발 - 수행결정나무(PDT) 방식을 사용하여 -

이향 국제한국언어문화학회 2021 한국언어문화학 Vol.18 No.1

지금까지 많은 연구들에서 담화 능력은 수험자의 의사소통 능력을 판단하기 위한 중요한 구인으로 다루어져 왔다. 하지만 담화 능력은 언어 능력의 다른 부분인 문법이나 어휘 혹은 발음 등과 같이 모국어 화자의 발화 직관만으로 평가 기준을 명확하게 기술하기 힘들다. 그래서 지금까지 많은 평가들에서는 담화 능력을 모호하고 주관적인 용어를 사용하여 채점 기준을 기술하고, 그 기준을 사용하여 평가해 왔다. 본 연구에서는 담화 능력을 평가하는 데 있어서 이러한 문제점을 보완하기 위하여 Fulture(2011)가 제안한 수행기반채점표(performance Decision Tree: PDT) 방식의 채점기준표 개발 방안을 제안하는 데 그 목적이 있다. 본 연구는 이해용․이향(2015)과 이용․이해용(2018)의 후속 연구로 선행연구들에서 다루었던 ‘부탁하기’ 과제에서의 모국어 화자의 담화 구조와 담화 능력별 한국어 학습자들의 담화 특징을 기반으로, 먼저 채점자들의 채점 경향을 분석하고, 그 결과를 범주화, 단계화하여 PDT 방식의 채점표를 개발하였다. 본 연구에서 제안한 담화 능력 평가를 위한 PDT 방식의 채점기준표는 담화 능력을 평가하는 데 있어 모호함이나 주관성으로 인하여 나타날 수 있는 채점 신뢰도 문제를 해결하여 보다 타당한 채점기준표를 개발하는 데 기여할 수 있을 거라 본다. Many researches have argued that discourse competence is a significant part of the communicative competences required for foreign learners. Accordingly, foreign language assessments treat discourse competence as an important construct for determining test takers' communicative competences. However, unlike for other linguistic competences, such as grammar, vocabulary, or pronunciation, it is difficult to present a clear grading scale for discourse competence that is based on the accuracy of native speakers on an absolute standard. As a result, many former assessments have used scoring rubrics with vague and subjective terms. To supplement these previous problems, this research presents a scoring rubric, based on the Performance Decision Tree method, and assesses native speakers’ discourse structures and foreign learners’ discourse characteristics per various levels of discourse competences in a REQUEST task. After conducting an additional analysis on graders’ grading tendencies, this study developed a performance-based scoring rubric by categorizing and sequencing the results. The Performance Decision Tree scoring rubric for discourse competence can contribute towards developing a more valid scoring rubric and solve the problem of scoring reliability, resulting from previous scoring rubrics’ vagueness and subjectivity.

Exploring Middle School Students’ Use of Scoring Rubrics for Improving English Writing

박지희,임현우 한국외국어대학교 외국어교육연구소 2020 외국어교육연구 Vol.34 No.2

The present study examines whether the use of a scoring rubric in writing instruction improves the writing performance of young inexperienced EFL learners. To explore the impact of different pedagogical assistances on students’ rubric use, 30 Korean middle school students were divided into two groups. One group was provided with a scoring rubric to study by themselves while the other group was given teacher instruction and modelling regarding how to use the rubric. Both groups performed the rubric-referenced assessment and revision of their initial English writing. In addition, four participants were invited to participate in teacher-guided revision conferences and their accounts explaining their understanding and use of the criteria in the rubric were analyzed. The results showed that providing a scoring rubric for students’ own use or giving one-time instruction of how to use it were both limited in helping these middle school students significantly improve their English writing. Pedagogical implications are provided along with the study’s limitations.

Can disclosure of scoring rubric for basic clinical skills improve objective structured clinical examination?

채수진,Miran Kim,Ki Hong Chang 한국의학교육학회 2016 Korean journal of medical education Vol.28 No.2

Purpose: To determine whether disclosure of scoring rubric for objective basic clinical skills can improve the scores on the objective structured clinical examination (OSCE) in medical students. Methods: Clinical performance score results of one university medical students (study group, n=345) were compared to those of another university (control group, n=1,847). Both groups took identical OSCE exam. OSCE rubric was not revealed to the study group until they were in the last 2 years of medical school. Results: There was no significant difference between before and after disclosure of rubric. However, history taking and physical examination scores of the study group were lower than those of the control group before the disclosure of rubric. After disclosure of rubric, the scores were either unchanged or slightly increased in the control group. Trend analysis of scores demonstrated that history taking and physical examination scores after the disclosure were significantly increased in the study group for 2 years. Conclusion: This study revealed that disclosure of basic clinical skills rubric to medical students could enhance their clinical performance, particularly in history taking and physical examination scores.

말하기 평가 채점기준표(rubric)의 상세화 및 적용 사례 연구 -설득 목적의 과제를 중심으로-

최영인,박성석,정민주,백정이,장성민,김예원,허모아,음소현,민병곤 한국국어교육학회 2023 새국어교육 Vol.- No.134

Purpose: This Study was performed to enhance scoring reliability through the refinement of speaking assessment rubric of persuasive speaking task. Methods: The speaking responses were collected from the preliminary survey and main survey on second-year secondary school students. In each survey, the refined scoring rubric was applied by two different groups of scorers and to two different target student groups. Results: After examining the scoring results, all scoring criteria, which consist of three major categories and seven subcategories, demonstrated high reliability except for the ‘explicitly of claim’ of the second scoring. In both surveys, the consistency of inter-rater reliability (ICC) was confirmed. Conclusion: In this study, it was confirmed that by refining the scoring rubric, the inter-rater reliability can be improved. 목적: 설득 목적의 말하기를 수행하는 과제에 대한 말하기 평가 채점기준표(rubric)를 일정 수준으로 상세화하고 그것이 말하기 채점타당도에 미치는 영향을 탐색한다. 방법: 중학교 2학년 학생들을 대상으로 말하기 평가(예비 조사, 본 조사)를 실시하여 말하기 응답을 수집하였다. 채점자와 채점 대상을 모두 달리하여 1차 채점(예비 조사 채점)과 2차 채점(본 조사 채점)이 진행되었고, 이때 상세화한 채점기준표를 적용하였다. 각각의 채점이 끝난 후 채점자 간 신뢰도(ICC)를 확인하였다. 결과: 채점 결과, 3개 대범주와 7개 하위 범주로 구성된 채점기준 항목에서 2차 채점의 ‘주장의 명료성’을 제외하고 모두 양호한 신뢰도를 보였다. 채점자 간 신뢰도가 낮은 항목에 대한 원인을 탐색하였다. 결론: 본 연구에서 제안한 ‘채점기준표 상세화’가 채점자 간 신뢰도를 높이는, 나아가 평가타당도를 높이는 방안이 될 수 있음을 확인하였다.

What Should Scoring Scale Descriptors Look Like?

Nan-young Ji 한국응용언어학회 2012 응용 언어학 Vol.28 No.1

The present study is aimed at investigating whether the specification of scoring scale descriptors affects the reliability of Korean EFL raters, and to what degree this occurs. A comparison of two scoring scales with differing levels of specification was undertaken. 32 Korean EFL secondary school teachers rated four writing samples using the two scoring scales and responded to a questionnaire asking about their perceptions of the two scales. The data were analyzed using a multi-faceted Rasch analysis. It was found that more specific scales improved inter-rater reliability. More specific scales also contributed to preventing biased rating against criteria. However, they failed to narrow the gap between raters in terms of degree of severity and to increase individual raters’ consistency in scoring different writing samples. An analysis of the raters’ responses to the survey questionnaire revealed that the more specific scoring scales were preferred as it was more useful in deciding on a score. Conversely, they were criticized for being too complex and time-consuming to apply.

What Should Scoring Scale Descriptors Look Like?

지난영 한국응용언어학회 2012 응용 언어학 Vol.28 No.1

The present study is aimed at investigating whether the specification of scoring scale descriptors affects the reliability of Korean EFL raters, and to what degree this occurs. A comparison of two scoring scales with differing levels of specification was undertaken. 32 Korean EFL secondary school teachers rated four writing samples using the two scoring scales and responded to a questionnaire asking about their perceptions of the two scales. The data were analyzed using a multi-faceted Rasch analysis. It was found that more specific scales improved inter-rater reliability. More specific scales also contributed to preventing biased rating against criteria. However, they failed to narrow the gap between raters in terms of degree of severity and to increase individual raters’ consistency in scoring different writing samples. An analysis of the raters’ responses to the survey questionnaire revealed that the more specific scoring scales were preferred as it was more useful in deciding on a score. Conversely, they were criticized for being too complex and time-consuming to apply.

고등학생 탐구보고서의 채점 기준 개발 및 적용-‘콩나물 기르기’ 탐구 사례-

최원호,윤희숙 한국현장과학교육학회 2014 현장과학교육 Vol.8 No.2

In this study, we developed an assessment rubric for inquiryreport with high validity and reliability to improve inquiry skillwhich is one of main goal in science curriculum of Korea. First,we developed the draft of scoring rubric based on the relatedstudies, and improved it by checking the validity, the inter-rateragreement and the reliability of scoring rubric. Finally, weadded explanations to the scoring rubric to help scorerunderstand it. The score of the inquiry report after applying thescoring rubric was positively related to that of science writtentest. And also we found that students had lack of inquiry skillin controling variables, suggesting results, and writingconclusion. 본 연구의 목적은 과학과 교육과정의 주요한 목표인 탐구 능력 신장을 위해 타당도와 신뢰도가 높은 탐구 능력 평가 도구인 탐구보고서 채점 기준을개발하는 것이다. 먼저 선행연구를 바탕으로 탐구보고서 채점 기준 초안을 개발하고, 타당도, 채점자간 일치도를 점검하여 채점 기준을 수정하였다. 마지막으로 채점 기준에 대한 이해를 돕기 위해 채점 기준에 대한 설명을 추가하였다. 채점 기준을 적용하여 탐구보고서를 채점한 결과, 보고서 점수는과학 과목의 지필평가 점수와 양의 상관관계가 있었으며, 학생들은 변인통제, 결과 제시, 결론 작성 등의 탐구 능력이 부족함을 알 수 있었다.

PBL型授業における自己評価についての一考察 : ルーブリックを使用した他者評価との差および省察に注目して : PBL형 수업에 있어서 자기평가에 대한 일고찰 : 루브릭을 사용한 타인 평가와 차이 및 자기 성찰에 주목해서

高草木美奈(Takakusaki Mina) 한국일본교육학회 2021 한국일본교육학연구 Vol.26 No.3

韓国の大学ではコロナ禍により2020年1学期から非対面(オンライン)授業になり、絶対評価での成績に変わった。このため総括評価における教師の裁量が拡大された場合も少なくなく、特に先進教授法として推奨されているPBL型授業の場合、学生の自己評価を総括評価に採用する事例も目にするようになった。さらにはPBL型授業の特性からか絶対評価であるべきだという教師側の声も複数耳にするようになった。しかし、自己評価を総括評価に組み入れるべきか否かは非常に慎重を期すべきセンシティブな問題でもあり、ルーブリックの作成などの検討されるべき条件や学習環境が多くあるものの、まだまだPBL型授業の総括評価については考察されていないのが現状である。そこで本稿では学生参加型の評価を採用するか否かの検討材料とするため、2021年1学期に筆者が担当したPBL型授業のプレゼン発表に対する学生の自己および他者評価を基に定量調査を行った。量的面のみならず質的面においても活動を通したリフレクションおよびどのような学習効果(学び)があったのかを学生のコメントからテクストマイニングを使用した定性調査を行い考察する。本稿が研究対象とした授業では日本文化をテーマにし、筆者勤務校学科のディプロマポリシーである卒業論文を目指したプレゼン発表を行い、ルービックによる形成評価を実施し、グループごとの自己評価の平均、他者評価の平均を比較するためt検定を行った。その結果、13項目あるすべての評価において他者評価より自己評価の方が高く有意差が見られた。PBL型授業が今後の学習につなげられたかについての質的考察としては、評価ポイントをルーブリックを通して意識させることができ、さらに卒業論文という目標を意識させることができたという学習効果が見られたものの、量的考察からは自己評価を総括評価に組み込むことの方法を含め、さらなる検討が必要だと思われる。 From the first semester of 2020, all classes have been converted to absolute evaluation as all classes are conducted online due to the corona virus. At the same time, teachers discretion was expanded in the summative evaluation. In the case of PBL-type classes, there are cases in which students self-evaluation is adopted in the summative evaluation. However, whether self-assessment should be integrated into summative evaluation is a sensitive issue that needs to be carefully considered, and So far, there has not been sufficient consideration of summative evaluation based on PBL-type classes. In this study, as part of a review of the possibility of adopting student participatory evaluation, a quantitative analysis was conducted based on the results of students self and peer evaluation on their presentation project, a PBL-type class in charge of the researcher in the first semester of 2021. In addition, the students comments were qualitatively analyzed in order to check the learning effect as well as self-reflection through the activity. In the class of this study, a formative evaluation was performed using a scoring rubric for presentations on Japanese culture, and a t-test was performed to compare the average of self-evaluation and the average of peer evaluation for each group. As a result, the self-evaluation showed higher scores than the peer evaluation in all the 13-item evaluations in both the midterm and final exams, and there was a significant difference between the scores. In addition, through qualitative analysis, it was confirmed that the presentations performed in the PBL-type class stimulated learners motivation in terms of graduation requirements and that the use of the explicit scoring rubric could help learners understand the evaluation factors. did. However, from the results of the quantitative survey, it is considered that additional review is necessary to introduce self-evaluation into summative evaluation.

A Study on a Vulnerability Scoring System for Intelligent Buildings

Minsu Park,Haein Kang,Hyoil Han,kyoungwon Suh 한국차세대컴퓨팅학회 2022 한국차세대컴퓨팅학회 학술대회 Vol.2022 No.10

The Common Vulnerability Scoring System (CVSS) is a well-known framework for providing the characteristics of a vulnerability and a numerical score reflecting its severity. Since the existing CVSS has been developed for an information technology (IT) environment, it is not suitable for an intelligent building (IB) environment where OT Operational Technology (OT) and Internet of Things (IoT) are integrated. For example, compared to IT systems that prioritize confidentiality, OT systems prioritize safety and have a very long lifespan. Therefore, if a vulnerable component is exploited in an intelligent building environment, the impact may be different from that in the IT environment. This paper identifies problems that may occur when applying the traditional CVSS to an intelligent building environment. We also propose an effective method to extend the CVSS for an intelligent building environment using threat intelligence and rubric.