MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
June 2008
Greetings! 
 
I have worked with multiple-choice items and the committees who write them for many years. While computer-based testing enhances exam security, many people involved in certification exam programs still worry that candidates share information about what is on the exam. 
 
Ross Brown
Consistency of Common Item p-Values across Multiple Choice Exam Administrations

An often-voiced concern is that candidates memorize item content, and then share the information with colleagues. Candidates so forewarned of what is on a test may have an unfair advantage, and might be able to pass the test without having the required level of knowledge and skill. To determine if candidates are able to gain an unfair advantage in this way, a systematic study was conducted.
 
The study examined item p-values across exam administrations. The p-value is the percent of candidates who answer an item correctly. If the p-values of items that appear on more than one examination increase by less than 10% from one exam to the next, there is a low probability that individual candidates are benefiting from items being leaked.
 
The p-values of items used on consecutive exams were compared. The table below shows the results. For all five examinations, the majority of the common items did not show p-value increases of more than 10%. Furthermore, even though some of the tests included a large number of carryover items, the items that had higher p-values (easier) were a very small percentage of the total number of items on the second test.
 
Thus, the use of common items on certification examinations does not seem to increase the percent of candidates who answer the item correctly. This makes one wonder about the efficiency of the networks of individuals who supposedly memorize items during a test to share with their colleagues. Perhaps their memories are not terribly accurate, their colleagues are not helped by the information, or misinformation is actually transmitted. 


Exam

Number of Common Items

Number of items on test

Number of items with p-value increases of 10% or more on second test

Percent of Common items that got easier as a percentage of all items on the second test

Exam 1  

250

313

23 items

7%

Exam 2

53

144

6 items

4%

Exam 3

290

310

3 items

1%

Exam 4

52

150

3 items

2%

Exam 5

196

252

7 items

3%

 
Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650
 


Coming Rasch-related Events
Feb 26 - June, 2018Online Advanced course in Rasch Measurement Theory (D.Andrich), University of Western Australia, Perth, Australia, http://www.education.uwa.edu.au/ppl/courses
March 23, 2018, Fri.12th Annual UK Rasch User Group Meeting, Loughborough University, Loughborough, England, www.rasch.org.uk
April 10-12, 2018, Tues.-Thurs. Rasch Conference: IOMW, New York, NY, www.iomw.org
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 22 - 24, 2018, Tues.-Thur. EALTA 2018 pre-conference workshop (Introduction to Rasch measurement using WINSTEPS and FACETS, Thomas Eckes & Frank Weiss-Motz), https://ealta2018.testdaf.de
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 27 - 29, 2018, Wed.-Fri. Measurement at the Crossroads: History, philosophy and sociology of measurement, Paris, France., https://measurement2018.sciencesconf.org
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 25 - July 27, 2018, Wed.-Fri. Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences", www.promsociety.org
July 29 - August 4, 2018 Vth International Summer School `Applied Psychometrics in Psychology and Education`, Institute of Education at the Higher School of Economics, St. Petersburg, Russia, https://ioe.hse.ru/en/announcements/215681182.html
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Sept. 3 - 6, 2018, Mon.-Thurs. IMEKO World Congress, Belfast, Northern Ireland, www.imeko2018.org
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com