MEASUREMENT RESEARCH ASSOCIATES
TEST INSIGHTS
April 2010
Greetings,
 
The computer is a fascinating and all powerful machine that can be programmed to do just about anything. For certification, the question is how can the computer be programmed to make the examination more valid and reliable.


Mary E. Lunz, Ph.D.

Executive Director

Alternative Item Formats For Computer Based Testing
For certification, we commonly think of multiple choice items because they are generally accepted, easy to administer, familiar from the development perspective, easy to score, and convenient for testing larger samples of candidates.  This article explores some alternatives to standard four or five part multiple choice items that can be delivered with computer based testing.  All item formats have assets and liabilities which are noted. All of these formats are available at Pearson VUE.

Video clips can be used to show organ function or patient interaction. They have action and sound. It is easy to incorporate video clips into a standard multiple choice exam. They must meet a standard video format (MPEG-1). Sound, if any is heard through ear phones.

Medical animation requires the development of the graphic images which are associated with the questions or actions/manipulations being tested. The graphics can be programmed so that the candidate controls the actions and behavior of the characters. The development of the graphics can be expensive and requires a great deal of work by the subject matter experts. The candidate may have to learn how to operate the computer programming to make the graphic images perform the tasks being tested.

Short answer responses allow the candidate to offer their thoughts and ideas related to a specific topic or question. The candidate explains the rationale for his/her judgments, selected procedures, or problem solving strategies. The difficulty with the short answer responses, is that they must be graded by a content expert unless a long, complex, and expensive scoring algorithm is constructed and validated. No such scoring algorithm has been developed to date.

Extended multiple choice items allow the inclusion of many possible responses (e.g. 10 - 20) based on the theory that candidates require more knowledge, skill and judgment to sort through all of the possible alternatives. Often candidates are allowed to choose all appropriate responses for this type of item. In reality, we have often found that there are four or five responses that are frequently selected and the others are not selected by any candidates.

Hot spot technology allows candidates to respond by clicking on points in an image to answer the question. The mouse cursor becomes a crosshair shape when over a hot-spot image and a marker is placed at the center of the crosshair when the candidate clicks the mouse button. A simplistic example is asking the candidate to click on the capital of Texas with many cities in Texas shown on a map. It is parallel to clicking on the section of an x-ray that identifies the patient's problem. 

Zoom technology uses the pearsonvue:scalefactors to allow a candidate to zoom into and out of content within an image . For example, an image is displayed and the candidate scrutinizes the image, zooming (or scaling) the image between a predefined set of zoom levels. This would be appropriate for microscopic slides. However, there is the danger that the image may become pixilated when zoomed to a very high scaling factor.

Finally, almost any type of visual can be used in the exams. More creative use of visuals could enhance the knowledge and skill tested in the exam.  

What can be done with computer programming is limited only by our imaginations and budgets.  The use of alternate item formats raises potential scoring, expert commitment, and programming cost issues. The question is "what is gained vs. what is lost?"  While the enhanced computer administration formats increase validity, what might be lost with regard to reliability or accuracy of candidate measurement?  Is the financial investment worth the long term convenience of less travel and "away" time for both candidates and examiners.  For example, if examiners score short answer responses at home instead of administering an oral examination at a specified place and time, will there be an impact on candidate outcomes when examiner colleagues are no longer available for professional and social interaction.
Measurement Research Associates, Inc.
505 North Lake Shore Dr., Suite 1304
Chicago, IL  60611
Phone: (312) 822-9648     Fax: (312) 822-9650


Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse Rasch Measurement Theory Analysis in R, Wind, Hua Applying the Rasch Model in Social Sciences Using R, Lamprianou Journal of Applied Measurement
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Rasch Models for Measurement, David Andrich Constructing Measures, Mark Wilson Best Test Design - free, Wright & Stone
Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias Diseño de Mejores Pruebas - free, Spanish Best Test Design A Course in Rasch Measurement Theory, Andrich, Marais Rasch Models in Health, Christensen, Kreiner, Mesba Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen
Rasch Books and Publications: Winsteps and Facets
Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes Rasch Models for Solving Measurement Problems (Facets), George Engelhard, Jr. & Jue Wang Statistical Analyses for Language Testers (Facets), Rita Green Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind Rasch Measurement: Applications, Khine Winsteps Tutorials - free
Facets Tutorials - free
Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:
Please email inquiries about Rasch books to books \at/ rasch.org

Your email address (if you want us to reply):

 

FORUMRasch Measurement Forum to discuss any Rasch-related topic

Coming Rasch-related Events
May 17 - June 21, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 12 - 14, 2024, Wed.-Fri. 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024
June 21 - July 19, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 5 - Aug. 6, 2024, Fri.-Fri. 2024 Inaugural Conference of the Society for the Study of Measurement (Berkeley, CA), Call for Proposals
Aug. 9 - Sept. 6, 2024, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 4 - Nov. 8, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 17 - Feb. 21, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 16 - June 20, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 20 - July 18, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com
Oct. 3 - Nov. 7, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com