Exploring the Role of Automatically-derived Text Complexity Features in L2 Reading Test Development

다국어 초록 (Multilingual Abstract)

Although automatic text analysis tools are available, little research has been conducted on the application of such tools in reading assessments. When the ratio of academic vocabulary and transitions are computed automatically and used in test development, the text selection-revision procedure can be fast and transparent by complementing test developers’ expertise. To obtain empirical evidence for the utility of automatic text complexity features, this study attempted to explore the role of automatically-derived text complexity features in an intensive English program (IEP) reading assessment. Based on previous literature and the testing context, a total of 11 text complexity features as lexical, syntactic, and semantic variables were chosen, and their accountability for the IEP reading item difficulty was automatically measured by using three text analysis tools—Lexile, the Compleat Lexical Tutor, and Coh-Metrix. Results showed that seven complexity features significantly correlated with the reading item difficulty. Stepwise multiple regressions showed that a set of four lexical and semantic text complexity features (i.e., word length, total word counts, Lexical Semantic Analysis (LSA), connectives) explained about 45% of the variance in the reading item difficulty. The results and findings of this study are discussed with regard to limitations and implications for both reading assessments and instruction.

참고문헌 (Reference)

1 전문기, "코메트릭스(Coh-Metrix)를 이용한 중학교 1학년 개정 영어 교과서의 코퍼스 언어학적 비교 분석" 팬코리아영어교육학회 21 (21): 265-292, 2009

2 전문기, "자동화된 언어분석 프로그램을 활용한 중학교 영어교과서의 학년 간 연계성 분석" 현대영어교육학회 16 (16): 195-218, 2015

3 김재은, "고등학교 영어 교과서, EBS 수능 연계 교재, 대학수학능력시험의 코퍼스기반 난이도 비교 분석" 한국멀티미디어언어교육학회 18 (18): 59-92, 2015

4 Gray, W. S., "What makes a book readable?" University of Chicago Press 1935

5 Tabachnick, B. G., "Using multivariate statistics" Pearson Education, Inc 2007

6 Freedle, R., "The prediction of TOEFL reading item difficulty: Implications for construct validity" 10 (10): 133-170, 1993

7 Drum, P. A., "The effects of surface structure variables on performance in reading comprehension tests" 16 (16): 486-514, 1981

8 Cobb, T., "The Compleat Lexical Tutor"

9 Fisher, D., "Text complexity: Raising rigor in reading" International Reading Association 2012

10 Castello, E., "Text complexity and reading comprehension tests" Peter Lang 2008

11 McNamara, D. S., "Learning from text: Effects of prior knowledge and text coherence" 22 (22): 247-287, 1996

12 IBM Corp, "IBM SPSS Statistics for Windows, Version 22.0" IBM Corp 2013

13 Livingston, S. A., "Delta equating" Educational Testing Service 2013

14 Embretson, S. E., "Component latent trait models for paragraph comprehension tests" 11 (11): 175-193, 1987

15 National Governors Center for Best Practices, "Common Core State Standards for English language arts and literacy in history/social studies, science, and technical subjects: Appendix A: Research supporting key elements of the standards and glossary of key terms"

16 Graesser, A. C., "Coh-Metrix: Analysis of text on cohesion and language" 36 (36): 193-202, 2004

17 Crossley, S. A., "Assessing text readability using cognitively based indices" 43 (43): 475-492, 2008

18 McNamara, D. S., "Assessing reading in the 21th century: Aligning and applying advances in the reading and measurement sciences" R & L Education 89-116, 2012

19 Alderson, C. C., "Assessing reading" Cambridge University Press 2000

20 Fisher, D., "Addressing CCSSO anchor standard 10: Text complexity" 91 (91): 250-250, 2014