Rasch 모형을 적용한 검사동등화에서 가교검사를 위한 동형 문항의 활용 가능성

지은림(Chi, Eunlim),박소연(Park, Soyeon) 한국교육평가학회 2011 교육평가연구 Vol.24 No.4

본 연구는 검사 동등화에서 발생하는 문항 노출의 위험성을 줄이기 위한 방안으로 공통 문항으로 구성된 가교검사를 사용하는 동등화에서 공통 문항의 단일 세트를 사용하는 대신에, 동형의 유사한 문항 세트들을 공통 문항으로 활용할 수 있는 가능성을 검증하고자 했다. 검사 동등화를 위해 대학생 244명을 대상으로 실시된 적성검사 문항들을 가지고 두 개의 동등화 설계를 구성하였다. 〈설계 Ⅰ〉은 동일한 문항들로 구성된 가교검사를 사용하는 동등화인 한편, 〈설계 Ⅱ〉는 동형의 유사한 문항 세트 2개를 공통 문항으로 취급하여 동등화하는 방안이었다. Rasch 모형 공통문항 비동등집단 설계에 의해 검사 동등화를 〈설계 Ⅰ〉과 〈설계 Ⅱ〉에서 각각 실시한 후에, 두 가지 설계에 의한 동등화 결과들을 비교해보았다. 비교 결과는 〈설계 Ⅰ〉과 〈설계 Ⅱ〉가 모수치 추정과 문항 적합도에 있어서 유의미한 차이를 보이지 않는다는 것을 보여주었다. 결론적으로 본 연구는 동형의 문항들을 활용하여 공통 문항 동등화를 실시함으로써 문항 노출의 위험성을 줄일 수 있는 가능성을 제시해주었다. This study aims to confirm the possibility of using equivalent form items, instead of using the exactly same common items, for the anchor test in test equating in order to reduce the risk of exposing test items. For the study, a data set of an aptitude test which was administered to 244 university students was divided into two equating designs. The design I uses the same items, while the design II used the equivalent form items for the anchor test. Test equating was conducted separately for each design by applying the method of Rasch common item nonequivalent group. The outcomes of equating by two designs were compared in terms of the estimation of parameters (item difficulty and person ability) and item fits. The results showed that there was no significant difference between two designs for the estimation of parameters and item fits. The findings suggest that tests can be equated by constructing an anchor test with equivalent form items for the security of test items. Furthermore, the possibility of using equivalent form items is expected to enhance the efficiency of equating tests.

An analysis of the test items on text structures in high school English tests

김성애 한국응용언어학회 2010 응용 언어학 Vol.26 No.4

The aim of this study is to investigate how students’ comprehension of text structures is assessed in high school English tests. For that aim, the study analyzed 30 high school English tests, focusing on the frequency/proportion of the test items made on text structures and the types of tasks required by those items. In order to investigate the similarities and/or differences among the tests, the study also compared the tests for the three grades and for the same grade. The result shows that there is a great variation among the tests in the frequency of the test items on text structures, but the types of tasks asked by those items are quite limited and similar across all the tests. Based on the result, the study argues that the test items on text structures in high school English tests leave much room for improvement and teachers need help and guide in the construction of test items.

Rasch 모형을 통한 <독서와 문법> 평가 문항 분석

이정애(Lee Jeong-ae),심재우(Shim Jae-woo) 국어문학회 2016 국어문학 Vol.63 No.-

본 연구는 현실적 차원에서 여전히 중요성을 갖는 선택형 문항들을 Rasch 모형을 통해 분석함으로써 교사가 출제하는 평가 문항의 신뢰도, 학생들의 능력에 따른 평가 적절성 여부, 평가 문항에 대한 정보와 판단의 타당성 등을 기술하고자 하였다. 우선 Rasch 모형의 분석을 위해 J 시 인문계 고등학생 총 179명이 치른 <독서와 문법>의 평가 문항을 대상으로 Winsteps 3.68 소프트 프로그램을 사용하였으며 결과는 다음과 같다. 첫째, 이 평가지는 학생의 능력 신뢰도가 문항 난이도의 신뢰도에 비해 낮으며 이를 통해 시험문항들이 학생능력의 상위집단과 하위집단의 능력에 맞게 제작되지 못했음을 알 수 있다. 둘째, 평가의 적합도의 분석 결과, 점이연상관계수가 .20이하를 보이는 5개의 문항들은 국어 성적의 분포 양상에 기여하지 못하는 부적합 문항으로 판정하였다. 셋째, 문항 난이도가 1.5이며, 전체 문항의 33%가 너무 쉽게 출제되어 난이도상 적합하지 않았다. 넷째, 문법 영역은 ‘상, 중상, 중하, 하’의 난이도 위계를 고루 포함하나 5개의 문항은 문법 지식을 측정하는 비교적 어려운 문항으로 확인하였다. 반면에 독서 영역은 ‘상’에 해당하는 문항이 없으며 ‘중하’와 ‘하’에만 집중된 대체로 쉬운 문제였다. 본 연구가 갖는 의의는 Rasch를 통해 교사들에게 선택형문항에 대해 신뢰도 및 적합도, 문항난이도, 타당도 등을 판단할 수 있는 정보를 제공하며, 이를 바탕으로 객관적이고 발전적인 평가 방향을 모색할 수 있다는 점이다. The main objectives of this study were to investigate item reliability, subject ability, and item fit indexes based on Rasch modeling. In particular, this study analyzed a Korean mid-term test results collected from 179 second year high school students, who attended a local high school in a mid-sized Korean city. The following were observed by the application of Winsteps 3.68, which is a software program for Rasch analyses: firstly, the person reliability as well as separation index was low, suggesting that the test itself was not successful in separating students along the ability continuum; secondly, all the test items met Rasch expectations. However, five test items had low levels of discrimination and failed to discriminate high ability students from low ability students. Next, personal ability estimates tended to be higher than item difficulty estimates, indicating that items, in general, were too easy for the students. Only one item was rightly targeted against the highest ability students. Finally, the 13 grammar items were evenly distributed in the hierarchy of item difficulties, forming four groups of items that were categorized as "the most difficult items", "the second most difficulty items", "the third most difficult items" and "the fourth most difficult items". However, it was found that reading proficiency items were located only in "the third most difficult" and "the fourth most difficult" groups of items. In fact, all of those reading proficiency items were from the text covered in the textbook and most of those reading proficiency items consisted of items that asked students to find simple facts based on the text. One of the implications of this study is that test designers, including in-service teachers themselves, should reflect on the test-making practices so that valid and reliable items are used in achievement tests.

새로운 시대, 문법 평가 경험의 체계화 방향 - 문법 문항의 비평과 제작 경험을 중심으로 -

김규훈 ( Kim Kyoohoon ) 한국문법교육학회 2021 문법 교육 Vol.43 No.-

The purpose of this study is to find out a way to systematize of the grammar assessment research on a basis of the grammar assessment experience. To accomplish this purpose, this study arranges criteria of analyzing grammar test items, “grammar ability, language materials for grammar test, types of grammar test items” by reviewing the issues from existing researches of the grammar assessment. This study gathers and analyzes the ethnography for grammar assessment experiences of Korean teachers in a field. At grammar test items in the national achievement assessment and the academic test which could find out feedback of the grammar curriculum, the common test items of the national achievement assessment are criticized and the set test items of the academic test are considered. As a result, a tendency of the “grammar ability” is captured that between a designed grammar ability and a practiced grammar ability are different, and the grammar ability is regarded as a general cognition in specific grammar test items. In the “language materials for grammar test” especially a “text”, a necessity of authentic texts and an usefulness of grammar explanation texts are raised questions. In case of the “types of grammar test items”, the “application” of grammar knowledge is ambiguous because what domains of learners grammatical thoughts are related to that “application”. To improve these awareness of grammar assessment experience, this study puts emphasis on a work to enhance the “connection grammar ability with types of grammar test items”. That is, the two dimensional classification of test objectives for grammar need to be set up, especially the behavioral domain of grammar have to be materialized by grammar performative verbs in the Korean curriculum. And as the “language materials for grammar test”, this study devises a method for an authentic text, and suggests reconsideration of the use of grammar explanation texts.

일제 강점기를 내용으로 하는 평가 문항 분석과 역사상

오정현 역사교육학회 2011 역사교육논집 Vol.46 No.-

The purpose of this study is to find out how much the items of the test reflected the spirit of the Japan occupation times, exploitation and resistance. It is found that the items had content validity showing the appropriateness of attaining the goals, importance of the facts, and developing thinking skills. Students tend to simply memorize the facts frequently covered in the test,which can lead to the fixed and shallow historical image to them. Students also showed difficulties in the test items dealing with particular chapters,the questions without pictures, and the items of unfamiliar historical facts,or having attractive wrong answers. In order for the test items to help students to understand history, two things should be considered when making questions. First, when selecting question materials, we should try to include proper amount of the facts,for students not to distort historical facts with narrow interpretation. Also it is recommended to choose rather complicated materials to improve their thinking skills. Second, thinking of the item frequency, we should not lean too much towards particular chapters or periods, because it can give students the biased historical image who think the more frequent the items, the more important. The history about Japan occupation should be covered with rational judgment rather than emotional approach and the test should also include value and attitude evaluation. Thus, it is suggested that supplemented tests, such as essay tests and performance tests, should be applied in the test, besides multiple choice tests.

논문 : 일제 강점기를 내용으로 하는 평가 문항 분석과 역사상

오정현 ( Jeong Hyun Oh ) 역사교육학회 2011 역사교육논집 Vol.46 No.-

The purpose of this study is to find out how much the items of the test reflected the spirit of the Japan occupation times, exploitation and resistance. It is found that the items had content validity showing the appropriateness of attaining the goals, importance of the facts, and developing thinking skills. Students tend to simply memorize the facts frequently covered in the test, which can lead to the fixed and shallow historical image to them. Students also showed difficulties in the test items dealing with particular chapters, the questions without pictures, and the items of unfamiliar historical facts, or having attractive wrong answers. In order for the test items to help students to understand history, two things should be considered when making questions. First, when selecting question materials, we should try to include proper amount of the facts, for students not to distort historical facts with narrow interpretation. Also it is recommended to choose rather complicated materials to improve their thinking skills. Second, thinking of the item frequency, we should not lean too much towards particular chapters or periods, because it can give students the biased historical image who think the more frequent the items, the more important. The history about Japan occupation should be covered with rational judgment rather than emotional approach and the test should also include value and attitude evaluation. Thus, it is suggested that supplemented tests, such as essay tests and performance tests, should be applied in the test, besides multiple choice tests.

영어 시험 문항 순서 차이에 따른 연습 혹은 반복 효과 분석

이연숙 한국영어평가학회 2021 영어평가 Vol.16 No.1

This study aims to examine the practice effects that can occur when multiple items assessing the same knowledge are presented on a in-school test that is usually based on a specific textbook with designated units and thus can have multiple items asking for the same knowledge/skill. For this purpose, the study analyzes the response data from two college English grammar tests administered in two semesters. The grammar tests were composed of two different sets (A and B) that had different item orders. The focus of the analysis was on the pairs of items that measure the same skill and appear in the opposite order in the two test sets, A and B. Applying the chi-square test to find out if there was any statistically significant different proportion correct (item facility) on each of these items across different test sets. The study found that the response data from one semester had more of these items with different item facility across the test sets, which indirectly implies the practice effect of these items. The paper concludes with suggestions for future research based on the limitations of the study.

문항반응이론을 활용한 한국어능력시험의 문항 양호도 분석-한국어능력시험 제28회~제32회를 중심으로-

장은아 국제한국어교육학회 2014 한국어 교육 Vol.25 No.4

The purpose of this study is to analyze the quality of items on the 28th through 32nd Test of Proficiency in Korean(TOPIK) using item response theory. Item Response Theory is invariable and insusceptible to the subject and test tool, not of the classical Test Theory that has been generally used in analyzing the difficulty or discrimination of items. The study found that there existed test items that were at an extremely high level or low level of difficulty, and their distribution was not constant amongst different tests and levels. Even in terms of item discrimination, it was found that there existed test items with little or low level of discrimination. In addition, it was shown that the difficulty level in certain types of test items from intermediate-level listening and some free-response test items from intermediate-to-advanced level writing, was either too high or low, and their discrimination level was found to be low. Therefore, measures to raise the quality of those relevant items must be sought out, and a new check on the rubric for making and grading the test questions is deemed necessary especially for the free-response writing questions. TOPIK is scheduled to be reshuffled and implemented with the new system starting from the year 2014. Tests will be implemented six times on a yearly basis. With the increase in the number of tests annually, chances are high that securing the quality of test items will become even more difficult. To tackle this problem, a conversion from the existing method of making test items to the item bank method using item response theory is inevitable.

An Analysis of the Multiple-Choice Test Items Constructed by Middle School English Teachers

김성애 한국응용언어학회 2009 응용 언어학 Vol.25 No.2

The aim of this study is to improve the quality of the multiple-choice test items constructed by middle school English teachers. For the aim the study collected 15 in-house English tests from 15 middle schools and analyzed the weaknesses of the items, focusing on three aspects: content, structure and options. The types and frequency of the weaknesses found in each test were described with examples and discussion. According to the results, the multiple-choice test items constructed by the middle school English teachers contain not a few weaknesses and as such, they cannot be considered to be a fully valid tool to measure the ability we want to foster in our students. Based on the results, it is argued that the in-house multiple-choice English tests as currently administered in the middle schools cannot contribute very much to achieving the beneficial backwash and that it is necessary to help the teachers develop the skills required for the construction of more valid test items.

An Analysis of the Test Items on Text Structures in High School English Tests

Sung-Ae Kim 한국응용언어학회 2010 응용 언어학 Vol.26 No.4

The aim of this study is to investigate how students’ comprehension of text structures is assessed in high school English tests. For that aim, the study analyzed 30 high school English tests, focusing on the frequency/proportion of the test items made on text structures and the types of tasks required by those items. In order to investigate the similarities and/or differences among the tests, the study also compared the tests for the three grades and for the same grade. The result shows that there is a great variation among the tests in the frequency of the test items on text structures, but the types of tasks asked by those items are quite limited and similar across all the tests. Based on the result, the study argues that the test items on text structures in high school English tests leave much room for improvement and teachers need help and guide in the construction of test items.