MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
June 2008
Greetings! 
 
I have worked with multiple-choice items and the committees who write them for many years. While computer-based testing enhances exam security, many people involved in certification exam programs still worry that candidates share information about what is on the exam. 
 
Ross Brown
Consistency of Common Item p-Values across Multiple Choice Exam Administrations

An often-voiced concern is that candidates memorize item content, and then share the information with colleagues. Candidates so forewarned of what is on a test may have an unfair advantage, and might be able to pass the test without having the required level of knowledge and skill. To determine if candidates are able to gain an unfair advantage in this way, a systematic study was conducted.
 
The study examined item p-values across exam administrations. The p-value is the percent of candidates who answer an item correctly. If the p-values of items that appear on more than one examination increase by less than 10% from one exam to the next, there is a low probability that individual candidates are benefiting from items being leaked.
 
The p-values of items used on consecutive exams were compared. The table below shows the results. For all five examinations, the majority of the common items did not show p-value increases of more than 10%. Furthermore, even though some of the tests included a large number of carryover items, the items that had higher p-values (easier) were a very small percentage of the total number of items on the second test.
 
Thus, the use of common items on certification examinations does not seem to increase the percent of candidates who answer the item correctly. This makes one wonder about the efficiency of the networks of individuals who supposedly memorize items during a test to share with their colleagues. Perhaps their memories are not terribly accurate, their colleagues are not helped by the information, or misinformation is actually transmitted. 


Exam

Number of Common Items

Number of items on test

Number of items with p-value increases of 10% or more on second test

Percent of Common items that got easier as a percentage of all items on the second test

Exam 1  

250

313

23 items

7%

Exam 2

53

144

6 items

4%

Exam 3

290

310

3 items

1%

Exam 4

52

150

3 items

2%

Exam 5

196

252

7 items

3%

 
Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650
 


Coming Rasch-related Events
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
Jan. 22-24, 2018, Mon-Wed. In-person workshop: Rasch Measurement for Everybody en español (A. Tristan, Winsteps), San Luis Potosi, Mexico. www.ieia.com.mx
April 10-12, 2018, Tues.-Thurs. Rasch Conference: IOMW, New York, NY, www.iomw.org
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 27 - 29, 2018, Wed.-Fri. Measurement at the Crossroads: History, philosophy and sociology of measurement, Paris, France., https://measurement2018.sciencesconf.org
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 25 - July 27, 2018, Wed.-Fri. Pacific-Rim Objective Measurement Symposium (PROMS), (Preconference workshops July 23-24, 2018) Fudan University, Shanghai, China "Applying Rasch Measurement in Language Assessment and across the Human Sciences" www.promsociety.org
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Sept. 3 - 6, 2018, Mon.-Thurs. IMEKO World Congress, Belfast, Northern Ireland www.imeko2018.org
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 22 - 24, 2018, Tues.-Thur. EALTA 2018 pre-conference workshop (Introduction to Rasch measurement using WINSTEPS and FACETS, Thomas Eckes & Frank Weiss-Motz), https://ealta2018.testdaf.de