Constructing Examinations From Calibrated Items

For certification and licensure, content experts work with a psychometric staff to develop test items and, from them, construct examinations. In some cases, the content experts know and/or care little about statistical analysis, making it crucial that psychometric information be presented to them in a form which is understandable, informative and non-threatening. Variables defined by the locations of calibrated items, supplemented with item content descriptions, have been useful for this purpose.

The Rasch model constructs a variable by locating each item on a logit scale oriented to extend from most to least difficult. It is fairly easy to attach a brief description of item content, task and taxonomy to each item calibration, and then to sort these annotated items by difficulty order within subtests. The printout of these sorted items, spaced out to represent the item calibrations, constructs a variable map detailing item difficulty and content within each subtest (or the entire item pool). From this, examinations can be constructed to meet desired test specifications and exam difficulty.

The abbreviated variable map shown in the Figure illustrates the use of this technique for examination development. The content and calibrated difficulty are provided for each item. The items are listed in ascending order of difficulty. An advance of more than 0.1 logits between adjacent items is indicated by a blank line. The variable map summarizes the availability of items across tasks, content and taxonomy areas in the subtest.

The variation of item difficulty calibrations within each subtest is easily seen. For instance, the committee might not have otherwise noticed that the SCOA items are easy, but the ELD items are on the harder side. Similarly, the Taxonomy 3, "problem solving", items are generally either very easy or very hard. All these considerations are easily overlooked when the committee merely select items from a long list of items, probably arranged by bank accession item number, or, if they are lucky, by task.

Certification requires accuracy at the decision point. Selecting items which are close to the decision point increases decision certainty by decreasing the error of measurement at that point. It is useful, therefore, to construct examinations with items "targeted" around the decision point (MPS: "minimum passing score" in Figure). Item difficulty and person ability are calibrated onto the same scale, so the ability measure that represents the decision point (MPS) can be designated on the itemized variable and the committee can see where the difficulties of the items selected for the examination stand in relation to the MPS.

It becomes obvious when unproductive items at the easy or hard ends of the variable are selected instead of items close to the MPS. Content experts do not need to be psychometrically oriented to see the difficulty level of the examination being constructed, and the number of items near the MPS.

Finally, the variable map is useful for identifying items needing revision. The items calibrated at the extreme ends of the variable can be seen to be the least useful for making decisions. Content experts can review these extreme items and revise them in order to make them more or less challenging as needed. The results of their revisions can be verified by the revised items' subsequent changes in position on the variable. If items representing a content, task or taxonomy area are scarce, that also becomes obvious and new items can be written to fill the gaps. In this example, it is clear that the item density is thin around the MPS, but that there are several very easy items.

Adding the ability distribution of a typical candidate group to the variable map would further increase its usefulness. This distribution reminds the committee what is the expected range of competency of those likely to sit the examination. This also encourages the committee to exclude items that are much too easy or too hard. At the same time, it facilitates the construction of an examination containing items targeted at all relevant ability levels, allowing almost all candidates to feel that at least some of the test items gave them the opportunity to demonstrate their own level of proficiency.

Variable maps can be constructed for subtests, examinations or entire item pools. The Rasch model provides the frame of reference and the item calibrations. The measurement system enables examinations to be constructed to match the desired standard and test specifications.

Mary Lunz
American Society of Clinical Pathologists

Figure for Constructing Examinations from Calibrated Item Variables

Constructing Examinations From Calibrated Items. Lunz M. … Rasch Measurement Transactions, 1989, 3:2 p.56-57

Rasch Books and Publications
Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, 2nd Edn. George Engelhard, Jr. & Jue Wang	Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene	Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver	Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone	Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes	Statistical Analyses for Language Testers (Facets), Rita Green	Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind	Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M	Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind	Rasch Measurement: Applications, Khine	Winsteps Tutorials - free Facets Tutorials - free	Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre	Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan
Other Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free	An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse	Rasch Measurement Theory Analysis in R, Wind, Hua	Applying the Rasch Model in Social Sciences Using R, Lamprianou	El modelo métrico de Rasch: Fundamentación, implementación e interpretación de la medida en ciencias sociales (Spanish Edition), Manuel González-Montesinos M.
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar	Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch	Rasch Models for Measurement, David Andrich	Constructing Measures, Mark Wilson	Best Test Design - free, Wright & Stone Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias	Diseño de Mejores Pruebas - free, Spanish Best Test Design	A Course in Rasch Measurement Theory, Andrich, Marais	Rasch Models in Health, Christensen, Kreiner, Mesba	Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen

Forum

Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

The URL of this page is www.rasch.org/rmt/rmt32c.htm

Website: www.rasch.org/rmt/contents.htm