MEASUREMENT RESEARCH ASSOCIATES TEST INSIGHTSDecember 2008
 Greetings!   After having worked with many tests over the years, the impact of measurement error on test reliability has become extremely clear.  It is also my observation that the quality, rather than the number of items, is the key to reducing measurement error and improving test reliability.   Surintorn Suanthong, Ph.D. Manager, Test Analysis and Research
Reliability for Multiple Choice Examinations

There are several ways to calculate the reliability of multiple choice examinations. Each formula means something slightly different with regard to the precision with which candidates are measured.

KR-20, Cronbach Alpha and Rasch Separation reliability are all estimates of
"true person variance / observed person variance" for the sample. KR-20 is based on split-halves, Cronbach Alpha is based on analysis of variance. These two are identical for complete, dichotomous tests, and are based on raw scores. Rasch Separation reliability is computed from the Rasch measures and their respective standard errors. The biggest practical difference is that KR-20 or Cronbach Alpha includes extreme items that are calculated to have small error variance and may increase reliability. For Rasch candidate separation reliability, extreme item measures are estimated to have large errors associated with the measure that are likely to lower the reliability.

Regardless of the formula used, the impact of the error of measurement on the variance of the candidates' scores is the key to understanding reliability. Generally, the larger the error of measurement is, the lower the reliability.  Measurement error is affected by 1) the number of items, 2) statistical performance of the items, and 3) quality, clarity, and relevance of the items. Therefore, the number of items on the test is not the sole factor that contributes to the reliability estimate.

The table shows results from five different multiple choice examinations. The number of items on the exam, in part relates to the error of measurement. However, the key is the amount of standard deviation relative to the error of measurement. When the standard deviation is higher relative to the error of measurement, the estimated reliability is higher, but when the error of measurement is higher relative to the standard deviation as in Exam 5, the estimated reliability is lower. The goal is to reduce the error of measurement so that differences among the candidates are accurately measured. In the end, it all depends on the quality of the items, rather than the number of items on the exam.

 Exam Number of Items Error of Measurement* Standard Deviation* Reliability Exam 1 211 .17 .65 .93 Exam 2 288 .14 .51 .92 Exam 3 171 .17 .56 .90 Exam 4 236 .18 .47 .85 Exam 5 235 .15 .31 .76

*presented in logits

 Measurement Research Associates, Inc. 505 North Lake Shore Dr., Suite 1304 Chicago, IL  60611 Phone: (312) 822-9648     Fax: (312) 822-9650   www.MeasurementResearch.com

Coming Rasch-related Events
Feb 26 - June, 2018Online Advanced course in Rasch Measurement Theory (D.Andrich), University of Western Australia, Perth, Australia, http://www.education.uwa.edu.au/ppl/courses
March 23, 2018, Fri.12th Annual UK Rasch User Group Meeting, Loughborough University, Loughborough, England, www.rasch.org.uk
April 10-12, 2018, Tues.-Thurs. Rasch Conference: IOMW, New York, NY, www.iomw.org
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 22 - 24, 2018, Tues.-Thur. EALTA 2018 pre-conference workshop (Introduction to Rasch measurement using WINSTEPS and FACETS, Thomas Eckes & Frank Weiss-Motz), https://ealta2018.testdaf.de
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 27 - 29, 2018, Wed.-Fri. Measurement at the Crossroads: History, philosophy and sociology of measurement, Paris, France., https://measurement2018.sciencesconf.org
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 25 - July 27, 2018, Wed.-Fri. Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences", www.promsociety.org
July 29 - August 4, 2018 Vth International Summer School `Applied Psychometrics in Psychology and Education`, Institute of Education at the Higher School of Economics, St. Petersburg, Russia, https://ioe.hse.ru/en/announcements/215681182.html
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Sept. 3 - 6, 2018, Mon.-Thurs. IMEKO World Congress, Belfast, Northern Ireland, www.imeko2018.org
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com