Constructing Examinations From Calibrated Items

For certification and licensure, content experts work with a psychometric staff to develop test items and, from them, construct examinations. In some cases, the content experts know and/or care little about statistical analysis, making it crucial that psychometric information be presented to them in a form which is understandable, informative and non-threatening. Variables defined by the locations of calibrated items, supplemented with item content descriptions, have been useful for this purpose.

The Rasch model constructs a variable by locating each item on a logit scale oriented to extend from most to least difficult. It is fairly easy to attach a brief description of item content, task and taxonomy to each item calibration, and then to sort these annotated items by difficulty order within subtests. The printout of these sorted items, spaced out to represent the item calibrations, constructs a variable map detailing item difficulty and content within each subtest (or the entire item pool). From this, examinations can be constructed to meet desired test specifications and exam difficulty.

The abbreviated variable map shown in the Figure illustrates the use of this technique for examination development. The content and calibrated difficulty are provided for each item. The items are listed in ascending order of difficulty. An advance of more than 0.1 logits between adjacent items is indicated by a blank line. The variable map summarizes the availability of items across tasks, content and taxonomy areas in the subtest.

The variation of item difficulty calibrations within each subtest is easily seen. For instance, the committee might not have otherwise noticed that the SCOA items are easy, but the ELD items are on the harder side. Similarly, the Taxonomy 3, "problem solving", items are generally either very easy or very hard. All these considerations are easily overlooked when the committee merely select items from a long list of items, probably arranged by bank accession item number, or, if they are lucky, by task.

Certification requires accuracy at the decision point. Selecting items which are close to the decision point increases decision certainty by decreasing the error of measurement at that point. It is useful, therefore, to construct examinations with items "targeted" around the decision point (MPS: "minimum passing score" in Figure). Item difficulty and person ability are calibrated onto the same scale, so the ability measure that represents the decision point (MPS) can be designated on the itemized variable and the committee can see where the difficulties of the items selected for the examination stand in relation to the MPS.

It becomes obvious when unproductive items at the easy or hard ends of the variable are selected instead of items close to the MPS. Content experts do not need to be psychometrically oriented to see the difficulty level of the examination being constructed, and the number of items near the MPS.

Finally, the variable map is useful for identifying items needing revision. The items calibrated at the extreme ends of the variable can be seen to be the least useful for making decisions. Content experts can review these extreme items and revise them in order to make them more or less challenging as needed. The results of their revisions can be verified by the revised items' subsequent changes in position on the variable. If items representing a content, task or taxonomy area are scarce, that also becomes obvious and new items can be written to fill the gaps. In this example, it is clear that the item density is thin around the MPS, but that there are several very easy items.

Adding the ability distribution of a typical candidate group to the variable map would further increase its usefulness. This distribution reminds the committee what is the expected range of competency of those likely to sit the examination. This also encourages the committee to exclude items that are much too easy or too hard. At the same time, it facilitates the construction of an examination containing items targeted at all relevant ability levels, allowing almost all candidates to feel that at least some of the test items gave them the opportunity to demonstrate their own level of proficiency.

Variable maps can be constructed for subtests, examinations or entire item pools. The Rasch model provides the frame of reference and the item calibrations. The measurement system enables examinations to be constructed to match the desired standard and test specifications.

Mary Lunz
American Society of Clinical Pathologists

Figure for Constructing Examinations from Calibrated Item Variables

ITEM VARIABLE FOR MICROBIOLOGY
(Subtest: Fungus)
Item
No.
Task* Content
Description
Taxonomy** Item
Calibration
Perceived
Difficulty
5l6
 
341
 
224
520
474
217
378
584
335
196
188
203
216
479
652
209
201
357
457
 
707
362
646
706
 
478
377
 
705
198
 
709
 
648
 
219
 
645
SCOA
 
DFC
 
CCLD
CCLD
CLD
CLD
CLD
ISOP
SM
CLD
CCLD
DFC
SCOA
ELD
DFC
DFC
CLD
DFC
DFC
 
CCLD
CCLD
DFC
CLD
 
SM
ELD
 
ELD
CLD
 
CLD
 
DFC
 
SM
 
CCLD
infec detection - yeast
 
micro morphology - mold
 
infec pathology - yeast-candida
mic-mac morphology - yeast
mic-mac morphology - mold-aspergillus
id procedure - dimorph-sporothrix
macro morphology - dermatophyte
incubation - dimorph
micro morphology - mold
micro morphology - mold-penicillium
mic-mac morphology - dimorph
micro morphology - dimorph-blasto
incubation - dermatophyte-genl
mic morphology - dermat-trichophyton
antigen detect - yeast
micro morphology - mold
micro morphology - dermatophytes
micro morphology - dimorph
incubation - dimorph
 
infection detec - yeast
micro morphology - dimorph-histoplasma
micro-mac morph - mold
mic-mac morphology - mold
 
culture - yeast
micro morphology - dimorph
 
micro morphology - yeast
macro morphology - dematiaceous
 
mic mac morph - mold
 
micro morphology - dermatophyte
 
growth requirement - yeast
 
micro morphology - dimorph
3
 
1
 
3
2
2
2
2
1
1
2
3
1
1
2
1
1
2
1
1
 
2
2
1
2
 
3
2
 
2
2
 
2
 
1
 
1
 
3
-3.28
 
-1.97
 
-0.81
-0.72
-0.67
-0.55
-0.45
-0.37
-0.26
-0.15
-0.08
0.02
0.09
0.12
0.18
0.24
0.29
0.30
0.36
 
0.53
0.59
0.67
0.76
 
0.89
0.92
 
1.07
1.11
 
1.33
 
2.00
 
2.51
 
2.65
Too
Easy
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
MPS
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Too
Hard
* Task: SCOA - Selected Course of Action; DFC - Define Fundamental Characteristic;
CCLD - Correlate Clinical and Laboratory Data; CLD - Correlate Laboratory Data;
ELD - Evaluate Laboratory Data; ISOP - Identify Standard Operating Procedure;
SM - Select Method
** Taxonomy: 1 = Recall; 2 = Interpretation; 3 = Problem Solving


Constructing Examinations From Calibrated Items. Lunz M. … Rasch Measurement Transactions, 1989, 3:2 p.56-57



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
May 17 - June 21, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 12 - 14, 2024, Wed.-Fri. 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024
June 21 - July 19, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 5 - Aug. 6, 2024, Fri.-Fri. 2024 Inaugural Conference of the Society for the Study of Measurement (Berkeley, CA), Call for Proposals
Aug. 9 - Sept. 6, 2024, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 4 - Nov. 8, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 17 - Feb. 21, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 16 - June 20, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 20 - July 18, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com
Oct. 3 - Nov. 7, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

The URL of this page is www.rasch.org/rmt/rmt32c.htm

Website: www.rasch.org/rmt/contents.htm