Comparing Measures: Scatterplots

Comparison of measures is frequently required. Item calibrations are compared when tests are equated to investigate stability. Person measures are compared to investigate change. Here are some techniques of comparison.

A necessary step before final conclusions can be drawn is to establish and maintain the frame of reference. Suppose the pre-test is Form E and the post-test is Form H. Simply subtracting a mean ability measure at pre-test from a mean ability at post-test is misleading as a measure of change for many reasons. The most obvious is that pre-test measures are relative to Form E and post-test measures are relative to Form H. The two Forms must be equated into a single frame of reference, i.e., with a shared origin and scale. Only then can valid comparisons between pre- and post-test measures be made. This equating may be an iterative procedure in which the following techniques can help.

A. Numerical comparisons. A.1 Direct comparisons:
If Bn is the measure of subject n, and Bm the measure of subject m, the obvious comparison is Bn-Bm. If SEn and SEm are the standard errors of Bn and Bm, then the SE of Bn-Bm = sqrt((SEn^2+SEm^2). A two-tailed test of the null hypothesis "Bn-Bm=0" is rejected at the .05 level when
|Bn-Bm| >= 1.96*sqrt((SEn^2+SEm^2)
>= 2.8 * SEn when SEn = SEm,
and at the .01 level when
|Bn-Bm| >= 2.58*sqrt((SEn^2+SEm^2)
>= 3.7 * SEn when SEn = SEm.

A.2 Normative comparisons:
It is often of interest to discover whether, as a group changes, individuals are changing faster or slower than their group. Suppose B1 is the measure of a subject at time 1, when the mean of the N measures is G1, and B2 is the measure of the same subject at time 2, when the group mean is G2. Then the gain of the subject relative to the group is (B2-G2)-(B1-G1). The error variances of the () terms have the form
SEBG1^2 = SEB1^2 + SDG1^2/(N-1).
The SE of the gain = sqrt((SEBG1^2+SEBG2^2).

A.3 Comparisons requiring rescaling:
Tests X and Y contain some equivalent items, but, due to different candidate behavior, rating scale specifications or analytical decisions, the logit units of the measurement scales estimated for the two tests have different lengths.

Suppose there are C items common to both tests. Their mean and variance in Test X are MX and VX, and in Test Y, MY and VY. Then the rescaling factor to place measures on Test Y into the scale of Test X is
SCALEYX = sqrt((VX/VY)
so that
B'Y = (BY - MY)*SCALEYX + MX
with
SE'Y = SEY*SCALEYX
Comparisons can now be made between BX and B'Y in the metric of Test X.

A.4 Evaluation of group stability:
Suppose there are C common elements in tests X and Y. Whether to attribute the inevitable differences in pairs of measures solely to measurement error is a statistical question. A test of the null hypothesis that the differences are attributable to measurement error is the homogeneity chi-square:

chi-square


B. Graphical comparisons.
B.1 Graphing without common scaling:
The numerical calculations can be summarized in a scatterplot of measures and confidence bands. Each element has two measures, BX and BY, and two standard errors, SEX and SEY.

Plotting confidence intervals is a two-stage process. For conventional 95% two-tailed confidence bands, the pairs of points are located at {UX, UY} for the upper confidence band and {LX, LY} for the lower band, where

Avge X Y


Drawing confidence bands


When you plot the {UX, UY} points, they will form a curved cloud (Figure 1). The points are exact, but provide too much detail. To expedite conceptualization, draw a smooth curve through these points with a thick felt-tip pen or computer drawing tool. Then erase the points. Repeat this for the {LX,LY} points. This draws the 95% confidence bands. Now plot your {BX,BY} data points (see Figure 2 (in printed text)). The identity line goes through {MEANX+AVGEX, MEANY+AVGEY}.

Identifying statistical outliers


B.2 Graphing with local scaling:
In Figure 2, the best fit line goes through points {MEANX+AVGEX-SDX, MEANY+AVGEY-SDY} and {MEANX+AVGEX+SDX, MEANY+AVGEY+SDY}. If this forms a noticeable angle with an identity line, then plot the confidence bands with the X and Y axes separately scaled. The locally scaled confidence bands are plotted in the same way as in B.1, but using these values:

Mean X Y


The best fit line now falls evenly between the bands.


Comparing measures: scatterplots. Luppescu S. … Rasch Measurement Transactions, 1995, 9:1 p.410



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
May 17 - June 21, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 12 - 14, 2024, Wed.-Fri. 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024
June 21 - July 19, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 5 - Aug. 6, 2024, Fri.-Fri. 2024 Inaugural Conference of the Society for the Study of Measurement (Berkeley, CA), Call for Proposals
Aug. 9 - Sept. 6, 2024, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 4 - Nov. 8, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 17 - Feb. 21, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 16 - June 20, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 20 - July 18, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com
Oct. 3 - Nov. 7, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

The URL of this page is www.rasch.org/rmt/rmt91c.htm

Website: www.rasch.org/rmt/contents.htm