MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
December 2008
Greetings!
 
After having worked with many tests over the years, the impact of measurement error on test reliability has become extremely clear.  It is also my observation that the quality, rather than the number of items, is the key to reducing measurement error and improving test reliability.
 
Surintorn Suanthong, Ph.D.
Manager, Test Analysis and Research
 
Reliability for Multiple Choice Examinations

There are several ways to calculate the reliability of multiple choice examinations. Each formula means something slightly different with regard to the precision with which candidates are measured.
 
KR-20, Cronbach Alpha and Rasch Separation reliability are all estimates of
"true person variance / observed person variance" for the sample. KR-20 is based on split-halves, Cronbach Alpha is based on analysis of variance. These two are identical for complete, dichotomous tests, and are based on raw scores. Rasch Separation reliability is computed from the Rasch measures and their respective standard errors. The biggest practical difference is that KR-20 or Cronbach Alpha includes extreme items that are calculated to have small error variance and may increase reliability. For Rasch candidate separation reliability, extreme item measures are estimated to have large errors associated with the measure that are likely to lower the reliability.
 
Regardless of the formula used, the impact of the error of measurement on the variance of the candidates' scores is the key to understanding reliability. Generally, the larger the error of measurement is, the lower the reliability.  Measurement error is affected by 1) the number of items, 2) statistical performance of the items, and 3) quality, clarity, and relevance of the items. Therefore, the number of items on the test is not the sole factor that contributes to the reliability estimate. 
 
The table shows results from five different multiple choice examinations. The number of items on the exam, in part relates to the error of measurement. However, the key is the amount of standard deviation relative to the error of measurement. When the standard deviation is higher relative to the error of measurement, the estimated reliability is higher, but when the error of measurement is higher relative to the standard deviation as in Exam 5, the estimated reliability is lower. The goal is to reduce the error of measurement so that differences among the candidates are accurately measured. In the end, it all depends on the quality of the items, rather than the number of items on the exam.   

 

Exam

Number of Items

Error of Measurement*

Standard Deviation*

Reliability

Exam 1

211

.17

.65

.93

Exam 2

288

.14

.51

.92

Exam 3

171

.17

.56

.90

Exam 4

236

.18

.47

.85

Exam 5

235

.15

.31

.76

*presented in logits

 
Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650
 


Coming Rasch-related Events
July 30 - Nov., 2018Online Introduction to Classical and Rasch Measurement Theories (D.Andrich), University of Western Australia, Perth, Australia, http://www.education.uwa.edu.au/ppl/courses
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 25 - Feb. 22, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 28, 2019 On-line course: Understanding Rasch Measurement Theory (ACER), https://www.acer.org/professional-learning/postgraduate/Rasch
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada.https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada.www.aera.net/Events-Meetings/Annual-Meeting
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com