온라인 리뷰 분석을 통한 상품 평가 기준 추출: LDA 및 k-최근접 이웃 접근법을 활용하여

이지현(Ji Hyeon Lee),정상형(Sang Hyung Jung),김준호(Jun Ho Kim),민은주(Eun Joo Min),여운영(Un Yeong Yeo),김종우(Jong Woo Kim) 한국지능정보시스템학회 2020 지능정보연구 Vol.26 No.1

Product evaluation criteria is an indicator describing attributes or values of products, which enable users or manufacturers measure and understand the products. When companies analyze their products or compare them with competitors, appropriate criteria must be selected for objective evaluation. The criteria should show the features of products that consumers considered when they purchased, used and evaluated the products. However, current evaluation criteria do not reflect different consumers’ opinion from product to product. Previous studies tried to used online reviews from e-commerce sites that reflect consumer opinions to extract the features and topics of products and use them as evaluation criteria. However, there is still a limit that they produce irrelevant criteria to products due to extracted or improper words are not refined. To overcome this limitation, this research suggests LDA-k-NN model which extracts possible criteria words from online reviews by using LDA and refines them with k-nearest neighbor. Proposed approach starts with preparation phase, which is constructed with 6 steps. At first, it collects review data from e-commerce websites. Most e-commerce websites classify their selling items by high-level, middle-level, and low-level categories. Review data for preparation phase are gathered from each middle-level category and collapsed later, which is to present single high-level category. Next, nouns, adjectives, adverbs, and verbs are extracted from reviews by getting part of speech information using morpheme analysis module. After preprocessing, words per each topic from review are shown with LDA and only nouns in topic words are chosen as potential words for criteria. Then, words are tagged based on possibility of criteria for each middle-level category. Next, every tagged word is vectorized by pre-trained word embedding model. Finally, k-nearest neighbor case-based approach is used to classify each word with tags. After setting up preparation phase, criteria extraction phase is conducted with low-level categories. This phase starts with crawling reviews in the corresponding low-level category. Same preprocessing as preparation phase is conducted using morpheme analysis module and LDA. Possible criteria words are extracted by getting nouns from the data and vectorized by pre-trained word embedding model. Finally, evaluation criteria are extracted by refining possible criteria words using k-nearest neighbor approach and reference proportion of each word in the words set. To evaluate the performance of the proposed model, an experiment was conducted with review on ‘11st’, one of the biggest e-commerce companies in Korea. Review data were from ‘Electronics/Digital’ section, one of high-level categories in 11st. For performance evaluation of suggested model, three other models were used for comparing with the suggested model; actual criteria of 11st, a model that extracts nouns by morpheme analysis module and refines them according to word frequency, and a model that extracts nouns from LDA topics and refines them by word frequency. The performance evaluation was set to predict evaluation criteria of 10 low-level categories with the suggested model and 3 models above. Criteria words extracted from each model were combined into a single words set and it was used for survey questionnaires. In the survey, respondents chose every item they consider as appropriate criteria for each category. Each model got its score when chosen words were extracted from that model. The suggested model had higher scores than other models in 8 out of 10 low-level categories. By conducting paired t-tests on scores of each model, we confirmed that the suggested model shows better performance in 26 tests out of 30. In addition, the suggested model was the best model in terms of accuracy. This research proposes evaluation criteria extracting method that combines topic extraction using LDA and refinement with k-near

보육시설 평가인증지표와 유치원 평가지표에 대한 비교 연구

하정희(Ha, Jeong-Hee) 한국변형영유아교육학회 2011 변형영유아교육연구 Vol.5 No.1

영유아교육기관에 대한 국가수준의 평가가 이제 제2주기를 맞고 있다. 수업 및 기관의 질적 향상을 위한 제도이므로 평가에 직접 영향을 미칠 수 있는 평가지표에 대한 논의가 필요하다. 본 연구는 보육시설의 평가인증 지표와 유치원 평가 지표를 비교함으로써 평가 지표에 대한 논의를 확장시키고자 하였다. 비교 결과, 첫째 보육시설 평가지표가 70개 항목으로 보다 구체적인 지침을 제시하고 있으며 보육환경과 운영관리를 강조하고 있는 반면, 유치원 평가지표는 47개 평가요소로 교육과정과 교육환경을 강조하고 있다. 둘째, 유치원 평가지표는 교육목표와 평가를 강조하고 있는 반면, 보육시설 평가인증지표는 보육계획과 보육활동, 일상적 양육의 통합을 강조하고 있다. 셋째, 교육환경은 공간 구성과 자료의 발달적합성을 강조하고 있는 반면, 보육환경은 공간배치와 자료 안내를 강조하고 있다. 넷째, 유치원 평가지표는 건강・안전 교육을, 평가인증지표는 청결과 안전관리를 강조하고 있다. 마지막으로, 보육시설 평가인증지표는 운영관리를 구체적으로 안내하고 있는 반면, 유치원 평가지표는 교직원과 원장의 전문성을 강조하고 있다. Presently, the evaluation for Child Care Center Accreditation and Kindergarten Evaluation are going into their second cycle. Therefore, we need a discussion on the evaluation criteria, as the evaluation system should improve the quality of lessons, and the center and evaluation criteria have a direct influence on evaluation. This study is to extend the discussion by comparing the evaluation criteria of the Child Care Center Accreditation with the Kindergarten evaluation. The results of the study are as follows: First, the criteria of the Child Care Center Accreditation has 70 items, the criteria of Kindergarten evaluation has 47 items, so there are more items in the Child Care Center Accreditation than in the Kindergarten evaluation. The criteria of the Child Care Center Accreditation emphasize the environment and management domains. The other side, the criteria of Kindergarten evaluation emphasize the curriculum and environment domains. Second, in the curriculum domain, the criteria of the Child Care Center Accreditation emphasize the integration of education plans, education activities and daily care activities . While the criteria of the Kindergarten evaluation emphasize the education objectives and education evaluation. Third, in the environment domain, the criteria of the Child Care Center Accreditation emphasize the guidance of educational materials. While the criteria of Kindergarten evaluation emphasize the suitability of the educational materials. Fourth, in the health, nutrition and safety domain, the criteria of the Child Care Center Accreditation emphasize cleanliness and safety management. The other side, the criteria of Kindergarten evaluation emphasize health education and safety education. Finally, in the management domain, the criteria of the Child Care Center Accreditation guide the management of the Child Care Center concretely. While the criteria of the Kindergarten evaluation emphasize the professional ability of teachers and principles of the kindergarten.

ODA 평가기준 다양화에 관한 연구: 이해관계자 인식조사 분석을 중심으로

김영수,김윤호 국제개발협력학회 2022 국제개발협력연구 Vol.14 No.4

Purpose: The OECD DAC presents six evaluation criteria. However, this study searches for additional evaluation criteria so that it can allow ODA evaluators to select appropriate evaluation criteria depending on their evaluation purpose and ODA requirements. Originality: Korean government has mechanically applied the six criteria suggested by DAC in evaluating its ODA policy. However, this practice may not meet the various objectives of numerous Korean ODA stakeholders. Thus, it is important to look for additional ODA evaluation criteria which reflect the opinions of the ODA stakeholders. This research aims to make contribution to the field of ODA policy by identifying additional ODA evaluation criteria. Methodology: This research used ‘constituency approach’ of organization theory to find additional ODA evaluation criteria. Initially, seven new (additional) evaluation criteria were identified based on the literature on policy evaluation of public administration and on international debates regarding ODA evaluation reform. Then, the seven criteria were presented to relevant stakeholders (or constituents) in government, public agencies, academia/research institutes, and private sector to conduct survey on the adequacy of the standards. The survey data were analyzed by SPSS ANOVA method. Result: The ANOVA analyses narrowed down the initial seven additional evaluation criteria to five which consist of Korean-style ODA, mutual cooperation, ownership, diversity/equity/inclusion, and democracy. Conclusion and Implication: It is critical to utilize various ODA evaluation criteria to which the ODA constituents can consent. By doing so, ODA policy can adequately evaluate its projects which tend to have diverse objectives and stakeholders. Thus, it is urgently required for the Korean government to adopt new ODA evaluation system, and revise its manuals accordingly.

온라인 구매 후기에 나타난 소비자의 니트 의류 제품 평가 차원

지혜경 한국니트디자인학회 2011 패션과 니트 Vol.9 No.1

This study aims to find out the evaluative criteria of consumers on purchasing and wearing knit clothing and how these criteria are evaluated. This study is based on the studies which clarified that there exist differences in evaluative criteria according to a consumer’s buying process. For the purpose, this study concentrated on clarifying the criteria on the processes after knit clothing purchase. To search the evaluative criteria on the processes after knit clothing purchase, the contents of on-line reviews of knit clothing on internet shopping malls were analyzed. Internet shopping malls for gathering on-line review data were randomly selected among top 20 of woman's apparel shopping mal1 in “Rankeydotcom”(http://www.rankey.com), the internet shopping malls rank analyzing service enterprise. 300 on-line reviews about knit/sweater, cardigan, which were written on the review menu of the shopping malls, were collected. The results of the study were as follows: First, according to the results of analysis on evaluative criteria which consumers importantly consider on the process after knit clothing purchase, the evaluative criteria have 4 dimensions, aesthetic appreciation, practicality/maintenance, original properties of knit, and economics. Second, among the diverse criteria of consumers, color(14.52%), the wearing situation or the applicability of coordination with other apparel products(12.41%), width/length effects(11.94%) were mostly referred criteria. Subsequently, there were design(6.11%), thickness(5.73%), texture(5.44%), and detail(5.06%). The results of the study will help owners of internet fashion shopping malls to operate the properties of products and service by using the clothing evaluative criteria of consumers.

기본 교육과정 성취기준 재해석을 통한 평가준거 성취기준과 평가기준 개발 방향 탐색

강은영,박경옥,박남수,박은희,이은규,박지민,염지혜 한국지체.중복.건강장애교육학회 2020 지체.중복.건강장애연구 Vol.63 No.1

Purpose: The purpose of this study is to explore the direction of the development of achievement criteria and evaluation criteria by reinterpreting the achievement criteria presented in the 2015 basic curriculum as a national level curriculum. Method: Eight elementary and middle school special teachers and three professors who have conducted a number of studies in the curriculum and evaluation area have experiences in the case-development. Results: First, the characteristics of students subject to special education, application of an evaluation criteria and an evaluation criteria development period, and composition of achievement criteria by subject, and an understanding of the differences must be premised. Second, analysis of the 'construct' contained in the content elements and performance elements of the achievement criteria, the identification of the 'scope and hierarchy' level, the method of expressing the performance elements included in the achievement criteria should be considered. It is also necessary to reflect the responsiveness, development pools of criteria for achievement of evaluation criteria in consideration of the characteristics of special education students, and reconceptualize the evaluation criteria of the basic curriculum. Third, the development of evaluation criteria and evaluation criteria based on the achievement criteria of the basic curriculum is educational in that it serves as a performance indicator that confirms the responsibility of education for individual students by switching to an evaluation system that records the educational growth of students. Conclusion: First, the premise of development should be considered before the development of evaluation criteria, achievement criteria and evaluation criteria. Second, it is necessary to develop pools of evaluation criteria, taking into account the characteristics of achievement standards and special education students, and to reconceptualize the evaluation criteria. Third, the development of achievement criteria and evaluation criteria based on the achievement criteria of the basic curriculum is meaningful in the essential aspect of the evaluation of a student's development of growth and confirmation of the responsibility of education. 연구목적: 이 연구는 국가수준 교육과정으로서의 2015 특수교육 교육과정 기본 교육과정에 제시된 성취기준을 특수교육대상학생의 특수성을 고려한 재해석을 통해 평가준거 성취기준과 평가기준 개발의 방향을 탐색하는데 목적이 있다. 연구방법: 이를 위해 초・중등 특수교사 8인과 교육과정 및 평가 영역에서 다수의 연구를 수행한 교수 3인이 함께 사례개발 경험을 가지면서 총 5차에 거쳐 집중적인 심층 인터뷰 과정을 거쳤으며, 참여 관찰방법을 활용하여 결과를 분석하였다. 연구결과: 첫째, 기본 교육과정 성취기준의 재해석을 통한 평가준거 성취기준과 평가기준 개발에 있어서는 기본 교육과정 적용 특수교육대상학생의 특성, 평가준거 성취기준 및 평가기준 개발 시기, 교과별 성취기준의 구성 방식의 차이에 대한 이해가 전제되어야 한다. 둘째, 평가준거 성취기준 및 평가기준 개발에서는 성취기준의 내용요소 및 수행요소에 내포된 ‘구인’의 분석과 ‘범위와 위계’ 수준 확인, 성취기준에 포함된 수행요소를 표현하는 방식과 학생의 반응성 반영, 특수교육대상학생의 특성을 고려한 평가준거 성취기준 풀(Pool) 개발, 기본 교육과정의 평가기준에 대한 재개념화 작업이 필요하다. 셋째, 기본 교육과정 성취기준에 토대한 평가준거 성취기준 및 평가기준 개발은 학생의 교육적 성장을 기록하는 평가체계로의 전환과 개별 학생에 대한 교육의 책무를 확인하는 성과지표로서의 역할이라는 점에서 교육적 함의를 지닌다. 결론: 첫째, 평가준거 성취기준 및 평가기준 개발에 앞서 개발의 전제에 대해 고려해야 한다. 둘째, 성취기준 및 특수교육대상학생의 특성을 고려한 평가준거 성취기준 풀 개발과 평가기준의 재개념화가 필요하다. 셋째, 기본 교육과정 성취기준에 토대한 평가준거 성취기준 및 평가기준 개발은 학생의 성장 발달과 교육의 책무 확인이라는 평가의 본질적 측면에서 의의를 지닌다.

대학 산학협력의 평가준거 개발 연구

곽진영,김진모 한국농·산업교육학회 2023 농업교육과 인적자원개발 Vol.55 No.4

The purpose of this study was to develop evaluation criteria that can be used autonomously when promoting industry-university cooperation in four-year universities and to find out their validity and relative importance. To achieve the purpose of this study, first, the industry-university cooperation evaluation criteria were devised. Secondly, the validity of the proposed evaluation criteria was verified, and finally, the relative importance of the final evaluation criteria was calculated. Based on the review of previous studies, the industry-university cooperation evaluation criteria were devised with 12 evaluation areas, 28 evaluation items, and 59 evaluation indicators in the early stage, and the final evaluation criteria was confirmed with 12 evaluation areas, 26 evaluation items, and 57 evaluation indicators based on expert consultations and two Delphi surveys. The third Delphi survey used the Analytical Hierarchy Process (AHP) method to evaluate the relative importance of each evaluation criterion as determined by all the panel assessors. The main conclusions are as follows. First, the evaluation criteria for the basic factors that universities should consider in promoting industry-university cooperation were proven that it is quite valid as a result of statistical verification. Second, in the process of examining the validity and relative importance of the evaluation criteria, it was confirmed that the six directions of developing the evaluation criteria for industry-university cooperation were significantly implemented. Third, as a result of calculating the relative importance, it was found that the panel of experts in various groups put the most emphasis on the criteria for objectives or plans, putting weight on the situation “C” at the CIPP stage. Finally, based on these conclusions, suggestions were made on the follow-up research and utilization related to the industry-university cooperation evaluation criteria. 이 연구의 목적은 4년제 대학에서 산학협력을 자율적으로 평가하기 위한 평가준거를 개발하고 그 타당도와 상대적 중요도를 구명하는 데 있었다. 연구의 목적을 달성하기 위해 문헌연구를 토대로 대학 산학협력 평가준거 초안을 구안하고, 초안에 대한 타당도를 검증한 후, 확정된 평가준거에 대한 상대적 중요도를 도출하였다. 선행연구 고찰을 토대로 12개 평가영역, 28개 평가항목, 59개 평가지표로 대학 산학협력 평가준거 초안을 구안하고, 전문가 협의 및 두 차례의 델파이 조사에 근거한 타당화 과정을 거쳐 12개 평가영역, 26개 평가항목, 57개 평가지표로 평가준거 최종안을 확정하였다. 타당성을 확보한 대학 산학협력 평가준거 최종안의 평가영역, 평가항목, 평가지표에 대하여 계층화 분석법(AHP)을 적용한 세 번째 델파이 조사를 실시하여 상대적 중요도를 산정하였다. 연구결과에 따른 결론은 다음과 같다. 첫째, 대학 산학협력 추진 시 기본적으로 고려해야 하는 평가준거 최종안은 통계적 검증 결과 상당히 타당함이 확인되었다. 둘째, 평가준거의 타당도와 상대적 중요도를 검토하는 과정에서 확인된 대학 산학협력 평가준거 개발의 여섯 가지 방향은 유의미하게 구현된 것으로 확인되었다. 셋째, 상대적 중요도 산정 결과 다양한 그룹의 전문가 패널은 전체적으로 목표나 계획에 대한 준거를 가장 중시하는 것으로 나타나 CIPP 단계에서 상황(C)에 무게를 두는 것으로 나타났다. 마지막으로 해당 연구결과를 바탕으로 대학 산학협력 평가준거와 관련한 후속연구 및 활용과 관련하여 제언하였다.

Evaluation Criteria for Road Networks in Residential Areas

하오근,박동주,이강대,원재무 대한토목학회 2011 KSCE JOURNAL OF CIVIL ENGINEERING Vol.15 No.7

The objective of this study is to identify evaluation criteria for evaluating residential area road networks. To this end, existing studies regarding urban plan paradigms, neighborhood road network designs and road network evaluation criteria are reviewed,leading to the selection through first and second expert questionnaire surveys of 42 road network evaluation criteria. The road network evaluation criteria were divided into interregional road networks and regional road networks. To derive the evaluation criteria, the former were divided into motorways (main roads and auxiliary main roads) and public transportation, while the latter were divided into motorways (distributed roads, local roads), pedestrian ways and bicycle ways. Analytical Hierarchy Process (AHP)analysis of the weighted values of the evaluation criteria revealed the regional road networks to be more important than the interregional road networks, and the eco-friendly transportation networks such as pedestrian ways and bicycle ways to be more important than others. The road network evaluation criteria were divided into quantitative and qualitative indicators, and the methods for their scoring were presented by considering the characteristics of each evaluation criterion. Finally, the practicality of these evaluation criteria was investigated with a case study that examined a residential area redevelopment project of the “Jeonnong ·dapshim-ri rearrangement promotion”. In the evaluation of the road networks before and after implementation of the development project, the evaluation scores of the interregional road networks did not show any significant differences, while the evaluation scores of regional road networks did. This result indicated that the conditions of the regional road networks, including motorways (distributed roads, local roads), were considerably improved by the case region development project.

Comparison of the morphologic criteria (RECIST) and metabolic criteria (EORTC and PERCIST) in tumor response assessments: a pooled analysis

김홍덕,김범준,김형수,김정한 대한내과학회 2019 The Korean Journal of Internal Medicine Vol.34 No.3

Background/Aims: The Positron Emission Tomography Response Criteria in Solid Tumors (PERCIST) or European Organization for Research and Treatment of Cancer (EORTC) criteria are used to assess metabolic tumor responses. However, tumor responses have shown considerable discrepancies between the morphologic criteria (Response Evaluation Criteria in Solid Tumors [RECIST]) and metabolic criteria. We performed this pooled study to compare the RECIST and metabolic criteria in the assessment of tumor responses. Methods: Electronic databases were searched for eligible articles with the terms “RECIST,” “PERCIST,” or “EORTC criteria.” The level of concordance in the tumor responses between the two criteria was estimated using κ statistics. Results: A total of 216 patients were collected from eight studies comparing the RECIST and EORTC criteria. The agreement of tumor responses between the two criteria was moderate (κ = 0.447). Eighty-six patients (39.8%) showed disagreement: tumor response was upgraded in 70 patients and downgraded in 16 when adopting the EORTC criteria. The EORTC criteria significantly increased the overall response rate (53% vs. 28%, p < 0.0001). The agreement of tumor responses between the RECIST and PERCIST was deemed fair (κ = 0.389). Of 407 patients from nine studies, 181 (44.5%) showed a discrepancy: using the PERCIST, tumor response were upgraded in 151 patients and downgraded in 30. When adopting the PERCIST, the overall response rate was also significantly increased from 30% to 55% (p < 0.0001). Conclusions: This pooled analysis demonstrates that the concordance of tumor responses between the morphologic criteria and metabolic criteria is not excellent. When adopting the metabolic criteria instead of the RECIST, overall response rates were significantly increased.

AHP를 활용한 단체참가 해외전시회 평가기준 개선 연구

김봉석,이은영,김인영 한국무역전시학회 2019 무역전시연구 Vol.14 No.2

The purpose of this study is to improve the evaluation criteria of overseas exhibition so that the exhibition could be effectively selected for the supporting program. Firstly, the purpose of this study is to develop the evaluation criteria by examining the current evaluation criteria of exhibition. Second, based on the factors derived from the review, we tried to refine the evaluation factors by conducting questionnaire survey of experts who have been participated in the overseas exhibition supporting program and related in this industry. Third, AHP analysis was conducted based on the refined factors to bring out the systematic improvement of the evaluation criteria for group participation of overseas exhibition. In order to achieve the purpose of the study effectively, the evaluation factors were constructed based on the evaluation criteria of the domestic exhibition evaluation creteria and the abroad creteria. From these factors, we constructed an expert group panel and conducted four times of expert focus group interviews(FGI) to carry out the factors. Finally, we used the factors for the AHP analysis, and based on the factor weights we could improve the evaluation criteria. The result of this study is that we made it possible to properly application the exhibition’s evaluation criteria to the current situation of exhibition industry. In addition, it is meaningful that the existing evaluation criteria have been improved by developing the evaluation factors which includes the quantitative and qualitative items through the systematic analysis method. 본 연구의 목적은 첫째, 기존의 평가기준과 해외의 전시회 평가기준 등을 고찰하여 해외 전시회 평가기준 요인을 도출하는 것과 둘째, 도출된 요인을 바탕으로 전시산업분야 전문가 및 해외전시회에 참가했던 참가업체를 대상으로 전문가 설문조사를 실시해 요인을 정제하는 것과 셋째, 정제된 요인을 바탕으로 AHP분석을 실시해 최종적인 평가기준을 개발하고자 하였다. 연구의 목적을 효과적으로 달성하기 위하여 국내의 해외전시회 평가기준과 해외의 전시회 평가 기준 및 AHP 기법에 대한 선행연구 분석을 실시하였으며, 4차례에 걸친 전문가 인터뷰(FGI)를 통해 AHP 측정에 사용될 항목 33개가 도출되었다. 그리고 도출된 항목들을 바탕으로 전시산업전문가를 대상으로 AHP 조사를 실시하였다. AHP 분석을 통해 각 항목에 대한 상대적 가중치를 도출하였고, 도출된 가중치를 바탕으로 항목의 배점을 부여하여 기존 단체참가 해외전시회 평가기준을 수정하였다. 또한 배점을 부여한 평가기준에 따라 측정할 수 있는 방법을 대략적으로 제시하였으며 기존의 주관적이었던 평가방법을 최대한 객관성 있고 타당한 평가를 할 수 있도록 대체하였다. 본 연구는 정량적 평가기준과 정성적 평가기준을 모두 아우르는 평가지표를 도출하고, 기존의 단체참가 해외전시회 평가기준을 실제 전시회의 상황에 맞도록 항목을 구성하는데 중점을 두었다. 기존의 주관적이었던 평가기준 또한 최대한 객관적으로 평가할 수 있는 항목으로 평가기준을 개선하였다는 점에서 큰 의미가 있다. 전시산업과 관련된 곳에서 종사하는 전문가들을 대상으로 조사를 실시하며 평가 항목과 배점을 도출하였으나, 이를 실제로 적용하여 전시회를 선정하기 위해서는 세부적인 측정척도와 측정방법에 대한 논의가 필요하다.

제7차 교육과정 체제에 가정된 교수활동 분석을 통한 수업평가 기준 설정의 기초 연구(1)

강현석 ( Hyeon Suk Kang ),정재임 ( Chae Im Chong ) 경북대학교 중등교육연구소 2004 중등교육연구 Vol.52 No.2

This study is a preliminary research to establish criteria for instruction evaluation through the analysis of teaching activities proposed in the 7th Curriculum. The curriculum supposes particular teaching activities. Therefore this particular curriculum model assumes components of desirable teaching activities which are consistent with itself. The analysis of teaching activities is very crucial to establish criteria to evaluate teaching activities because components of desirable teaching activities mean criteria. As you see above, curriculum, instruction, and evaluation can be seen as an important chain to understand problems of school education. As a result of this study, the criteria for instruction evaluation consist of five broad levels, which are established to reflect the logic and practical aspects of instruction evaluation at once. The five levels of criteria for instruction evaluation are as follows: 1. The first level: Evaluation areas(Plans for instruction, The implementation and operation of instruction, The evaluation of instruction), 2. The second level: The components of evaluation (The area of plans for instruction, The implementation and operation of instruction, The evaluation of instruction, 3. The third level: The criteria for evaluation, 4. The fourth level: The sub-criteria for evaluation, 5. The fifth level: Rating scale (Kyungpook National Univ./Kyungpook Girls` High School)