총조사 오차(Total Survey Error)의 패러다임으로 이해하는 표본조사 = Understanding Sample Surveys with the Total Survey Error Paradigm

다국어 초록 (Multilingual Abstract)

Sample surveys have been widely practiced as a data collect tool for understanding the society and its population. More importantly, as surveys provide vital data sources for government official statistics, their quality may directly influence government policies, guiding the lives of many people. Therefore, it is important to understand and evaluate sample surveys systematically based on science. This paper provides a brief background and history of sample surveys and introduces the total survey error paradigm crucial for a scientific understanding of sample surveys.

국문 초록 (Abstract)

사회인구에 대한 이해의 목적으로 사용하는 데이터 수집을 위하여 표본조사가 널리 시행되고 있다. 특히 표본조사가 정부의 공식통계에 활발히 사용되고 있으므로, 표본조사의 품질이 일반국민의 생활을 결정짓는 정책 수립 등에 직접적인 영향을 미칠 수 있다. 따라서 표본조사의 품질을 과학을 바탕으로 하여 구조적으로 평가하는 노력이 필요하다. 본 원고는 표본조사의 간략한 역사와 더불어 표본조사 품질을 과학적으로 이해하는데 핵심적인 역할을 하는 총조사 오차(total survey error)의 패러다임을 소개하겠다.

참고문헌 (Reference)

1 김세용, "전화조사의 체계적 편향 - 2007년 대통령선거 여론조사들에 대한 메타분석 -" 한국통계학회 22 (22): 375-385, 2009

2 김서영, "응답 거부와 부재율이 무응답 오차에 미치는 영향: 경제활동인구조사를 중심으로" 한국통계학회 22 (22): 667-676, 2009

3 Hatchett, S., "White respondents and Race of Interview Effects" 39 : 523-528, 1975

4 Couper, M.P., "What They See Is What We Get: Response Options for Web Surveys" 22 (22): 111-127, 2004

5 Lee, S., "Weighting Telephone Samples Using Propensity Scores. in: Advances in Telephone Survey Methodology" John Wiley 170-186, 2007

6 Bethlehem, J.G., "Weighting Nonresponse Adjustments Based on Auxilary Information. in: Survey Nonresponse" Wiley 275-288, 2002

7 Kalton, G., "Weighting Methods" 19 : 81-97, 2003

8 Couper, M.P., "Web Surveys: A Review of Issues and Approaches" 64 : 464-494, 2000

9 Couper, M.P., "Web Survey Methods: Introduction" 72 : 831-835, 2008

10 Couper, M.P., "Visual Context Effects in Web Surveys" 71 : 91-112, 2007

11 Royall, R.M., "Variance Estimation in Finite Population Sampling" 73 : 351-358, 1978

12 Duncan, K.B., "Using Propensity Scores to Control Coverage Bias in Telephone Surveys" 27 (27): 121-130, 2001

13 Lee, S., "Understanding Respondent Driven Sampling from a Total Survey Error Perspective, Survey Practice"

14 Andersen, R., "Total Survey Error" Jossey-Bass 1979

15 Sudman, S., "Thinking about Answers: The Application of Cognitive Processes to Survey Methodology" Jossey-Bass 1996

16 Hastie, R., "Theoretical Issues in Person Memory. in: Person Memory: The Cognitive Basis of Social Perception" Erlbaum 1980

17 Cantor, D., "The Use of Monetary Incentives to Reduce Nonresponse in Random Digit Dial Telephone Surveys. in: Advances in Telephone Survey Methodology" John Wiley 471-498, 2008

18 Groves, R.M., "The Role of Topic Interest in Survey Participation Decisions" 68 (68): 2-31, 2004

19 Tourangeau, R., "The Psychology of Survey Response" Cambridge University Press 2000

20 Hansen, M.H., "The Problem of Nonresponse in Sample Surveys" 41 : 517-529, 1946

21 Yang, M-L, "The Interviewer Effect When There is an Education Gap with the Respondent: Evidence from a Survey on Biotechnology in Taiwan" 37 : 1321-1331, 2008

22 Singer, E., "The Impact of Privacy and Confidentiality Concerns on Census Participation" 57 : 465-482, 1993

23 Groves, R.M., "The Impact of Nonresponse Rates on Nonresponse Bias" 72 (72): 167-189, 2008

24 Schwarz, N., "The Impact of Administration Mode on Response Effects in Survey Measurement" 5 : 193-212, 2006

25 Beebe, T.J., "The Effects of Survey Mode and Asking about Future Intentions on Self-reports of Colorectal Cancer Screening, Cancer Epidemiology" 17 (17): 785-90, 2008

26 Schuman, H., "The Effects of Black and White Interviewers on Black Responses in 1968" 35 : 44-68, 1971

27 Curtin, R., "The Effect of Response Rate Changes on the Index of Consumer Sentiment" 64 (64): 413-428, 2000

28 Lee, S., "The Effect of Question Order on Self-Rated General Health Status in a Multilingual Survey Context" 169 (169): 1525-1530, 2009

29 Belson, W., "The Design and Understanding of Survey Questions" Gower Publishing 1981

30 Hyman, H. H., "The Current Status of American Public Opinion. in: The teaching of Contemporary Affairs: Twenty-first Yearbook of the National Council for the Social Studies" National Education Association 11-34, 1950

31 Rosenbaum, P.R., "The Central Role of the Propensity Score in Observational Studies for Causal Effects" 70 : 41-55, 1983

32 Kish, L., "Survey Sampling" John Wiley & Sons 1965

33 Converse, J. M., "Survey Research in the United States: Roots and Emergence 1890-1960" University of California Press 1987

34 Olson, K., "Survey Participation, Nonresponse Bias, Measurement Error Bias, and Total Bias" 70 : 737-58, 2006

35 Little, R.J.A., "Survey Nonresponse Adjustments for Estimates of Means" 54 : 139-157, 1986

36 Groves, R.M., "Survey Methodology" Wiley-Interscience 2004

37 Mathiowetz, N., "Survey Measurement of Work Disability: Summary of a Workshop" National Academy Press 2000

38 Groves, R.M., "Survey Errors and Survey Costs" Wiley 1989

39 Lee, S., "Statistical Estimation Methods in Volunteer Panel Web Surveys" University of Maryland 2004

40 Little, R.J.A., "Statistical Analysis with Missing Data, 2nd Edition" John Wiley 2002

41 Fowler, F.J., "Standardized Survey Interviewing; Minimizing Interviewer-Related Error" Sage Publications 1991

42 American Association for Public Opinion Research, "Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 6th edition" AAPOR 2009

43 Council of American Survey Research Organizations, "Special Report: On the Definition of Response Rates" CASRO 1982

44 Tourangeau, R., "Spacing, Position, and Order: Interpretive Heuristics for Visual Features of Survey Questions" 68 : 368-393, 2004

45 Phillips, D.L., "Some Effects of ‘Social Desirability’ in Survey Studies" 77 : 921-940, 1972

46 Heckathorn, D.D., "Social Network of Jazz Musicians. in: Changing the Beat: A Study of Worklife of Jazz Musicians, Vol III: Respondent-Driven Sampling: Survey Results by the Research Center for Arts and Culture" 48-61, 2003

47 Kreuter, F., "Social Desirability Bias in CATI, IVR, and Web Surveys: The Effects of Mode and Question Sensitivity" 72 : 847-865, 2008

48 Abadie, A., "Simple and Bias-Corrected Matching Estimators. Technical Report" Department of Economics 2002

49 Lynn, P., "Separating Refusal Bias and Non-contact Bias: Evidence from UK National Surveys" 51 (51): 319-333, 2002

50 Moore, J.C., "Self/Proxy Response Status and Survey Response Quality: A Review of Literature" 4 : 155-172, 1988

51 Schonlau, M., "Selection Bias in Web Surveys and the Use of Propensity Scores" 37 : 291-318, 2009

52 Krosnick, J.A., "Satisficing in Surveys: Initial Evidence" 70 : 29-44, 1996

53 Couper, M.P., "Risk of Disclosure, Perceptions of Risk, and Concerns about Privacy and Confidentiality as Factors in Survey Participation" 24 : 255-275, 2008

54 Groves, R.M., "Responsive Design for Household Surveys: Tools for Actively Controlling Survey Errors and Costs" 169 (169): 439-457, 2006

55 Krosnick, J., "Response strategies for Coping with the Cognitive Demands of Attitude Measures in Surveys" 5 (5): 213-236, 1991

56 Sudman, S., "Response Effects in Surveys" Aldine 1974

57 Javeline, D., "Response Effects in Polite Cultures: A Test of Acquiescence in Kazakjstan" 63 : 1-28, 1999

58 Heckathorn, D.D., "Respondent-Driven Sampling: A New Approach to the Study of Hidden Populations" 44 : 174-199, 1997

59 Heckathorn, D.D., "Respondent-Driven Sampling II: Deriving Valid Population Estimates from Chain-Referral Samples of Hidden Populations" 49 : 11-34, 2002

60 Oksenberg, L., "Research on the Design and Conduct of the National Medical Expenditure Survey Interviews, Final Report" Survey Research Center, Institute of Social Research 1992

61 Macarthur, C., "Reliability and Validity of Proxy Respondent Information about Childhood Injury: An Assessment of a Canadian Surveillance System" 145 : 834-841, 1997

62 Bethlehem, J.G., "Reduction of Nonresponse Bias through Regression Estimation" 4 : 251-260, 1998

63 Schwarz, N., "Rating Scales: Numeric Values May Change the Meaning of Scale Labels" 55 : 618-630, 1991

64 Meislin, R., "Racial Divisions Seen in Poll on Howard Beach Attack, New York Times, January 8"

65 Finkel, S.E., "Race-of-Interviewer Effects in a Pre-election Poll Virginia 1989" 55 : 313-330, 1991

66 Cotter, P.R., "Race-of-Interviewer Effects in Telephone Interviews" 46 : 278-284, 1982

67 Schuman, H., "Questions and Answers in Attitude Surveys" Academic Press 1981

68 Harkness, J., "Questionnaire Translation. in: Cross-Cultural Survey Methods" Wiley 35-56, 2002

69 Lee, S., "Propensity Score Adjustment as a Weighting Scheme for Volunteer Panel Web Surveys" 22 (22): 329-349, 2006

70 Crowder, R.G., "Principles of Learning and Memory" Lawrence Erlbaum 1976

71 Waugh, N.C., "Primary Memory" 72 : 89-104, 1965

72 Schuman, H., "Polls, Surveys, and the English Language" 6-7, 1997

73 Morin, R., "Polling-in Black and White, Washington Post, November 5"

74 Couper, M.P., "Picture This! An Analysis of Visual Effects in Web Surveys" 68 (68): 255-266, 2004

75 Lee, S., "Perceptions of Disability: The Effect of Self- and Proxy-Response" 20 (20): 671-686, 2004

76 Strack, F., "Order Effects in Survey Research: Activation and Information Functions of Preceding Questions. in: Context Effects in Social and Psychological Research" Springer 1992

77 Weeks, M.F., "Optimal Times to Contact Sample Households" 44 : 101-114, 1980

78 Bishop, G.F., "Opinions on Fictitious Issues: The Pressure to Answer Survey Questions" 50 : 240-250, 1986

79 Neyman, J., "On the Two different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection" 97 (97): 558-625, 1934

80 Peytchev, A., "Not All Survey Effort is Equal: Reduction of Nonresponse Bias and Nonresponse Error" 73 (73): 785-806, 2009

81 Brick, J.M., "Nonsampling Errors in Dual Frame Telephone Surveys"

82 Groves R.M., "Nonresponse in Household Surveys" John Wiley & Sons 1998

83 Groves, R.M., "Nonresponse in Household Interview Surveys" Wiley 1998

84 Merkle, D., "Nonresponse in Exit Polls: A Comprehensive Analysis. in: Survey Nonresponse" Wiley 243-259, 2002

85 Groves, R.M., "Nonresponse Rates and Nonresponse Error in Household Surveys" 70 : 646-675, 2006

86 Biemer, P.P., "Nonresponse Bias and Measurement Bias in a Comparison of Face to Face and Telephone Interviewing" 17 : 295-320, 2001

87 Srinath, K.P., "Multiple Sampling in Nonresponse Problems" 66 : 583-586, 1971

88 Valentine, C.A., "Missing Men. Report to the US Bureau of the Census" US Bureau of the Census, mimeo 1971

89 International Telecommunication Union, "Measuring the Information Society" International Telecommunication Union 2009

90 Groves, R.M., "Measuring and Explaining Interviewer Effects in Centralized Telephone Surveys" 50 : 251-266, 1986

91 Pascale, J., "Measuring Private and Public Health Coverage: Results from a Split-Ballot Experiment on Order Effects" U.S. Census Bureau 2001

92 Call, K.T., "Measuring Health Insurance Coverage in Surveys. State Health Access Data Assistance Center Survey Guidelines Series"

93 Heckman, J.J., "Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme" 64 : 605-654, 1997

94 Heckman, J.J., "Matching as an Econometric Evaluation Estimator" 65 : 261-294, 1998

95 Dillman, D.A., "Mail and Telephone Surveys: The Total Design Method" Wiley-Interscience 1978

96 Dillman, D.A., "Mail and Internet Surveys: The Tailored Design Method. 2nd Edition" John Wiley 2000

97 Lee, S., "Linguistic Minorities and Nonresponse Error" 72 : 470-486, 2008

98 Groves, R.M., "Leverage-Saliency Theory of Survey Participation: Description and an Illustration" 64 : 299-308, 2000

99 Durbin, J., "Inversions and Rank Correlation Coefficients" 13 : 303-309, 1951

100 Kane, E.W., "Interviewer Gender and Gender Attitudes" 57 : 1-28, 1993

101 Tucker, C., "Interviewer Effects in Telephone Surveys" 47 : 84-95, 1983

102 Davis, R.E., "Interviewer Effects in Public Health Surveys" 25 : 14-26, 2010

103 Suchman, L., "Interactional Troubles in Face-to-Face Survey Interviews" 85 : 232-253, 1990

104 Rice, M.F., "Information and Communication Technologies and the Global Digital Divide Technology Transfer, Development, and Least Developing Countries" 1 (1): 72-87, 2003

105 Kish, L., "Inference from Complex Samples(with Discussion)" 36 : 1-37, 1974

106 Curtin, R., "Incentives in Telephone Surveys: A Replication and Extension" 23 : 91-105, 2007

107 Garren, S.T., "Improved Ratio Estimation in Telephone Surveys Adjusting for Noncoverage" 28 (28): 63-76, 2002

108 Menon, G., "How Well Do You Know Your Partner? Strategies for Formulating Proxy-Reports and Their Effects on Convergence to Self-Report" 32 : 75-84, 1995

109 Groves, R.M., "Gender Effects among Telephone Interviewers in a Survey of Economic Attitudes" 14 : 31-52, 1985

110 Singer, E., "Exploring the Meaning of Consent: Participation in Research and Beliefs about Risks and Benefits" 19 : 273-285, 2003

111 Lee, S., "Exploring Nonresponse Bias in a Health Survey Using Neighborhood Characteristics from Census Data" 99 (99): 1811-1817, 2009

112 Tourangeau, R., "Everyday Concepts and Classification Errors: Judgments of Disability and Residence" 22 (22): 385-418, 2006

113 Brown, R.V., "Evaluation of Total Survey Error" 17 (17): 335-356, 1967

114 Schaeffer, N.C., "Evaluating Race-of-Interviewer Effects In a National Survey" 8 : 400-419, 1980

115 Langlet, E., "Estimation of Proxy, Time and Content Effect for the Health and Activity Limitation Survey" American Statistical Association 89-97, 2000

116 Lee, S., "Estimation for Volunteer Panel Web Surveys Using Propensity Score Adjustment and Calibration Adjustment" 37 : 319-343, 2009

117 Sudman, S., "Effects of Time and Memory Factors on Response in Surveys" 68 : 805-815, 1973

118 Link, M.W., "Effects of Survey Mode on Self-Reports of Adult Alcohol Consumption: A Comparison of Mail, Web and Telephone Approaches" 66 : 239-245, 2005

119 Dillman, D.A., "Effects of Questionnaire Length, Respondent-Friendly Design, and a Difficult Question on Response Rates for Occupant-Addressed Census Mail Surveys" 57 : 289-304, 1993

120 Groves, R.M., "Dual Frame, Mixed Mode Survey Design" 1 : 263-286, 1985

121 Schober, M.F., "Does Conversational Interviewing Reduce Survey Measurement Error?" 61 : 576-602, 1997

122 Norris, P., "Digital Divide: Civic Engagement, Information Poverty, and Internet Worldwide" Cambridge University Press 2001

123 Smith, T.W., "Developing Comparable Questions in Cross-National Surveys. in: Cross-Cultural Survey Methods" John Wiley & Sons 69-92, 2002

124 Krosnick, J.A., "Designing Good Questionnaires: Insights from Psychology" Oxford University Press 1998

125 Lyberg, L., "Data Collection Methods and Measurement Errors: An Overview. in: Measurement Errors in Surveys" John Wiley & Sons 237-258, 1991

126 Conrad, F.G., "Customizing Survey Procedures to Reduce Measurement Error. in: Cognition and Survey Resea" John Wiley & Sons 301-317, 1999

127 Schwarz, N., "Culture-Sensitive Context Effects: A Challenge for Cross-Cultural Surveys. in: Cross-Cultural Survey Methods" John Wiley & Sons 93-100, 2002

128 Korns, A., "Coverage Issues Raised by Comparisons between CPS and Establishment Employment, Proceedings of the Social Statistics Section" American Statistical Association 60-69, 1977

129 Terhanian, G., "Correcting Data from Online Surveys for the Effects of Nonrandom Selection and Nonrandom Assignment" Harris Interactive 2000

130 Schwarz, N., "Context Effects in Social and Psychological Research" Springer-Verlag 1991

131 Todorov, A., "Context Effects in National Health Surveys: Effects of Preceding Questions on Reporting Serious Difficulty Seeing and Legal Blindness" 64 : 65-76, 2000

132 Martin, E., "Context Effects for Census Measures of Race and Hispanic Origin" 54 : 551-566, 1990

133 Billiet, J.B., "Context Effects as Substantive Data in Social Surveys. in: Context Effects in Social and Psychological Research" Springer-Verlag 131-147, 1992

134 Keeter, S., "Consequences of Reducing Nonresponse in a Large National Telephone Survey" 64 : 125-148, 2000

135 Cannell, C.F., "Comparison of a Self-Enumerative Procedure and a Personal Interview: A Validity Study" 27 : 250-264, 1963

136 Fowler, F.J., "Comparing Telephone and Mail Responses to the CAHPS™ Survey" 37 (37): MS41-MS49, 1999

137 Tourangeau, R., "Cognitive Science and Survey Methods. in: Cognitive Aspects of Survey Methodology: Building a Bridge between Disciplines" National Academy Press 73-100, 1984

138 Belli, R.F., "Cognitive Psychology and Survey Methodology: Nurturing the Continuing Dialogue between Disciplines" 21 : 141-144, 2007

139 Tourangeau, R., "Cognitive Processes Underlying Context Effects in Attitude Measurement" 103 : 299-314, 1988

140 Schwarz, N., "Cognitive Dynamics of Proxy Responding: The Diverging Perspectives of Actors and Observers" 13 : 159-179, 1997

141 Jabine, T.B., "Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines" National Academy Press 1984

142 Tourangeau, R., "Cognitive Aspects of Survey Measurement and Mismeasurement" 15 (15): 3-7, 2003

143 Jobe, J.B., "Cognition and Survey Measurement: History and Overview" 5 : 175-192, 1991

144 Lenski, G.E., "Caste, Class, and Deference in the Research Interview" 65 : 463-467, 1960

145 Keeter, S., "Calling Cell Phones in ’08 Preelection Polls" News Release from The Pew Research Center for the People & the Press 2008

146 Weeks, M.F., "Call Scheduling with CATI: Current Capabilities and Methods. in: Telephone Survey Methodology" Wiley 403-420, 1988

147 Todorov, A., "Bias in Proxies' Reports of Disability: Data from the National Health Interview Survey on Disability" 90 : 1248-1253, 2000

148 Singer, E., "Attitudes and Behavior: The Impact of Privacy and Confidentiality Concerns on Participation in the 2000 Census" 65 : 368-384, 2003

149 Montaquila, J.M., "Aspects of Nonresponse Bias in RDD Telephone Surveys. in: Advances in Telephone Survey Methodology" John Wiley 561-586, 2008

150 Tourangeau, R., "Asking Sensitive Questions: The Impact of Data Collection Mode, Question Format, and Question Context" 60 : 275-304, 1996

151 Bradburn, N.M., "Answering Autobiographical Questions: The Impact of Memory and Inference on Surveys" 236 : 208-216, 1987

152 Locander, W., "An Investigation of Interview Method, Threat and Response Distortion" 71 : 269-275, 1976

153 Groves, R.M., "An Experimental Measurement of Total Survey Error" 1984

154 Krosnick, J., "An Evaluation of a Cognitive Theory of Response-Order Effects in Survey Measurement" 51 : 201-219, 1987

155 Hansen, M.H., "An Evaluation of Model-Dependent and Probability Sampling Inferences in Sample Surveys" 78 : 776-793, 1983

156 Brooks, C.A., "An Error Profile: Employment as Measured by the Current Population Surveys" Office of Federal Statistical Policy and Standards, US Department of Commerce 1978

157 Politz, A.N., "An Attempt to Get the ’Not at Homes’ into the Sample without Call-backs" 45 : 136-137, 1950

158 Politz, A.N., "An Attempt to Get the ’Not at Homes’ into the Sample without Call-backs" 44 : 9-31, 1949

159 Turner, C.F., "Adolescent Sexual Behavior, Drug Use, and Violence: Increased Reporting with Computer Survey Technology" 280 : 867-873, 1998

160 Cannell, C.F., "A Report on Respondents’ Reading of the Brochure and Letter and an Analysis of Respondents’ Level of Information" Institute for Social Research, University of Michigan 1965

161 U.S. Department of Commerce, "A Nation Online: How Americans Are Expanding Their Use of the Internet"

162 Bishop, G., "A Comparison of Response Effects in Self-Administered and Telephone Surveys. in: Telephone Survey Methodology" Wiley 321-340, 1988

163 Schonlau, M., "A Comparison Between a Propensity Weighted Web Survey and an Identical RDD Survey" 22 : 128-138, 2004