MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
February 2009
Greetings,
 
There is frequently a question about the number of items needed to cover the content and insure candidate separation reliability.  Our simple study revealed that the quality of items is as or more important than the absolute number of items on the test to achieve satisfactory reliability.
 
Mary E. Lunz, Ph.D.
Director

 
Test Length and Test Reliability for Multiple Choice Examinations

Some believe that longer multiple choice tests tend to be more reliable because more items automatically reduce the error of measurement.  Indeed, a sufficient number of items must be included to cover the content areas tested; however, there are other factors that contribute to how efficiently a test measures and separates candidate ability. 

The indication of reliability is candidate separation reliability, calculated as [SD2 - SE2/ SD2] using Rasch logit measures.  This reliability index is appropriate because the goal of any certification examination is to distinguish between those candidates who are worthy of passing and those who are not.  The better the test items distinguish among candidate abilities, the less measurement error there is in the examination and the higher the candidate separation reliability, regardless of the number of items.  For example, deleting poorly performing items makes the test shorter, but also increases the accuracy of measurement because the items producing the most error are eliminated from the examination. 

In this simple study, we looked at test length compared to test reliability. Six different exams were included. The table shows the tests in reliability order along with the number of items in each test.  The number of items and reliability do not appear to be related. A test of 211 items had a reliability of .93 while another test of 233 items had a reliability of .78.  It appears that the quality of the items is as important, or even more important, than the absolute number of items on the test. Of course, a reasonable number of items must be included to insure accurate measurement and content coverage.

   Table of Test Length and Reliability

Exam

N of items

Reliability

Exam 1

211

.93

Exam 2

133

.91

Exam 3

193

.90

Exam 4

129

.86

Exam 5

236

.85

Exam 6

233

.78


Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650



Coming Rasch-related Events
July 30 - Nov., 2018Online Introduction to Classical and Rasch Measurement Theories (D.Andrich), University of Western Australia, Perth, Australia, http://www.education.uwa.edu.au/ppl/courses
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 25 - Feb. 22, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 28, 2019 On-line course: Understanding Rasch Measurement Theory (ACER), https://www.acer.org/professional-learning/postgraduate/Rasch
April 4 - 8, 2019, Thur.-Mon. NCME annual meeting, Toronto, Canada.https://ncme.connectedcommunity.org/meetings/annual
April 5 - 9, 2019, Fri.-Tue. AERA annual meeting, Toronto, Canada.www.aera.net/Events-Meetings/Annual-Meeting
May 24 - June 21, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 28 - July 26, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 9 - Sept. 6, 2019, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 11 - Nov. 8, 2019, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 24 - Feb. 21, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - June 19, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 26 - July 24, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 7 - Sept. 4, 2020, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 9 - Nov. 6, 2020, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 25 - July 23, 2021, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com