Greetings
While the validity of any exam is really the accurate or
valid interpretation of exam scores, such interpretations depend upon the
content and constructs measured in the exam.
The appropriate knowledge and skill must be tested to achieve valid scores.
Tara McNaughton
Manager, Test Development and Analysis
|
Practice Analysis Issues and Methods
|
The primary purpose of certification exams is distinguishing
individuals capable of practicing safely and effectively from those who can
not. It is essential to ensure that
these high stakes examinations are valid, reliable, and objective measures of
candidate ability.
A valid test is one that measures what it's meant to
measure, with the underlying goal to accurately interpret scores. In order to make valid interpretations of
scores, examinations should have 1) face validity, 2) content validity and 3)
construct validity. Face validity simply implies that the test appears to cover
appropriate areas of practice. Content
and construct validity suggest that the test measures the knowledge and skills
pertinent to the field of practice. A
strong argument for content validity can be achieved by conducting a practice
analysis.
The test blueprint supports the construct and content
validity and is the basis for selecting items to be included on an exam. The blueprint of the exam should mirror
practice in the field not only in terms of the knowledge and skill sampled, but in
the proportion, depth, and breadth of coverage.
The test blueprint can be created to the
specification of current work in the field by means of a practice
analysis. The practice analysis solicits
information from practicing professionals in the field concerning what they
need to know, the tasks they perform, and the relative frequency or importance
of each. Multiple related tasks form a
content area on the test blueprint.
|
In the development, data collection, and
analysis of a practice analysis, the expertise of subject matter experts,
practitioners, and psychometricians is required. The subject matter experts often do the
initial development of the survey by identifying the areas of knowledge and
skill required to practice effectively.
Psychometricians can offer guidance in survey development. Practitioners may be involved in the early
stages of development through field tests or focus groups to further refine the
survey. Finally, a sample of
practitioners is selected as the survey respondents.
The practice analysis can emphasize tasks and
skills, as well as, the knowledge necessary to perform appropriately in the
field. The general format may be a list
of tasks followed by likert scale responses designed to measure the frequency
or importance of each for the professional in the field. Another perspective on developing the
practice analysis questionnaire is to focus on the knowledge, skills, and
abilities needed for safe and effective practice within the field. Developing this type of questionnaire focuses
more on the personal attributes of the professional, as well as, the tasks
being performed.
It can be difficult to encourage busy
professionals to take the time to answer the survey. Strategies to maximize response rates include
limiting the length of the survey to a page or two which focus on the most
critical issues. Some organizations do
follow ups such as multiple mailings, reminder phone calls and emails, or
offering incentives for completing the survey.
The final step is the analysis of the data and
the structuring of a usable test blueprint.
Psychometricians produce summary information concerning the measured
importance and frequency of each item on the survey. Subject matter experts review these results
and make refinements in the final product based on their knowledge and
experience in the field.
One interesting finding in the literature is
that different approaches to practice analyses can produce very similar
results. The quality of expert judgments
in developing and reviewing the analysis may influence the outcome more than
the method or expense used for the practice analysis. This underscores the importance of expert
judgments in determining the eventual test blueprint that is developed. While practitioners may be the best judges of
frequency, some research indicates that measures of criticality may be best
left to expert judgments.
|