## Reasonable mean-square fit values.

"Less control over the data → more off-dimensional behavior → more departures in the data from the Rasch model"

Though the ideal for measurement construction is that data fit the Rasch model, all empirical data departs from the model to some extent. But how much unmodelled noise is tolerable? Conventional statisticians base their decisions on significance tests, but these are heavily influenced by sample size.

"Very large samples form a special source of problems. This is because no model can ever be supposed to be perfectly fitted by data, so with a sufficiently large sample any model would have to be discarded. In connection with the problem Martin-Löf (1974) stated: 'This indicates that for large sets of data it is too destructive to let an ordinary significance test decide whether or not to accept a proposed statistical model [or data], because, with few exceptions, we know that we shall have to reject it even without looking at the data simply because the number of observations is so large. In such cases, we need instead a quantitative measure of the size of the discrepancy between the statistical model and the observed set of data.'" (Gustafson 1980).

Convenient quantitative measures of fit discrepancy are mean-square residual summary statistics, such as OUTFIT and INFIT. These statistics have expectation 1.0, and range from 0 to infinity. Mean-squares greater than 1.0 indicate underfit to the Rasch model, i.e., the data are less predictable than the model expects. Mean-squares less than 1.0 indicate overfit to the Rasch model, i.e., the data are more predictable than the model expects. A mean-square of 1.2 indicates that there is 20% more randomness (i.e., noise) in the data than modelled. A mean-square of 0.7 indicates a 30% deficiency in Rasch-model-predicted randomness (i.e., the data are too Guttman-like), which implies 100*(1-0.7)/0.7 = 43% more ambiguity in the inferred measure than modelled (e.g., the item difficulty estimated from low-ability persons differs noticeably from the item difficulty estimated from high-ability persons).

Reasonable Item Mean-square Ranges
for INFIT and OUTFIT
Type of Test Range
MCQ (High stakes)
MCQ (Run of the mill)
Rating scale (survey)
Clinical observation
Judged (agreement encouraged)
0.8 - 1.2
0.7 - 1.3
0.6 - 1.4
0.5 - 1.7
0.4 - 1.2

When is a mean-square too large or too small? There are no hard-and-fast rules. Particular features of a testing situation, e.g., mixing item types or off-target testing, can produce idiosyncratic mean-square distributions. Nevertheless, here, as a rule of thumb, are some reasonable ranges for item mean-square fit statistics. In Classical Test Theory, 3-PL IRT, and conventional statistics, low mean-squares are considered good. In Rasch theory, they indicate some redundancy in the responses, but they do no harm.

Ben Wright & John Michael Linacre

J.-E. Gustafson (1980) Testing and obtaining fit of data to the Rasch model. British Journal of mathematical and Statistical Psychology, 33, p.220.

P. Martin-Löf (1974) The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and observational data. Scandinavian Journal of Statistics, 1, 3.

Note by Linacre: Informal simulations studies and experience analyzing hundreds of datasets indicate that:

Interpretation of parameter-level mean-square fit statistics:
>2.0Distorts or degrades the measurement system
1.5 - 2.0Unproductive for construction of measurement, but not degrading
0.5 - 1.5Productive for measurement
<0.5Less productive for measurement, but not degrading.
May produce misleadingly good reliabilities and separations

The Mean-Square statistic is also called the Relative Chi-Square and the Normed Chi-Square. According to www.psych-it.com.au/Psychlopedia/article.asp?id=277, in Structural Equation Modeling, the criterion for acceptance varies across researchers, ranging from less than 2 (Ullman, 2001) to less than 5 (Schumacker & Lomax, 2004).

[Later:] Overfit (mean-squares less than 1.0).

• Mean-squares are forced to average near 1.0. So, if there are very large mean-squares (for instance, due to random lucky guessing), this forces there also to be low mean-squares.
Advice: always investigate and remove items with high mean-squares before looking at items with low mean-squares.
• Mean-squares less than 1.0 are too predictable. They do not contradict what we know, but they do not tell us much that is new about what we want to know. So overfitting items are inefficient.
i) if we are developing a new test: replace overfitting items with more efficient items.
ii) if we are analyzing data from an existing test: take no action, even an item with a very low mean-square tells us a little something that is new and useful. We don't want to waste any useful information.
• To reduce a large set of items to a small Rasch-compliant set:
1. Eliminate the underfitting items >1.2 - this removes the uncooperative items
2. Reanalyze the data. You will notice that the mean-square fit statistics recentralize.
3. Now eliminate the underfitting and overfitting items (>1.2 and <0.8) - this optimizes the selection of the reasonably behaved items.

See also "Size vs. Significance: Mean-Square and Standardized Chi-Square Fit Statistics" www.rasch.org/rmt/rmt171n.htm

Item Fit and Person Fit

From a statistical Rasch perspective, persons and items are exactly the same. They are merely parameters of the Rasch model. So the fit criteria would be exactly the same. But, from a substantive perspective, persons and items differ. We expect the items to be better-behaved than the persons. We also expect item difficulties to continue into the future, but we expect person abilities to change. Also, we expect items to be encountered by many, many persons, but persons to encounter relatively few items. Consequently, we are usually stricter in our application of fit rules to items than to persons. A few maverick persons in a dataset don't worry us - they will have negligible impact on anything else. But a few maverick items raise questions about test administration, data entry accuracy, the definition of the latent variable, etc. We will immediately focus our attention on them because they may be symptomatic of a more pervasive problem, such as the wrong key for a multiple-choice test, or reversed-coded items on a survey.

Reasonable mean-square fit values. Wright BD, Linacre JM … Rasch Measurement Transactions, 1994, 8:3 p.370

Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

 Forum Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
Sept. 27-29, 2017, Wed.-Fri. In-person workshop: Introductory Rasch Analysis using RUMM2030, Leeds, UK (M. Horton), Announcement
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Oct. 25-27, 2017, Wed.-Fri. In-person workshop: Applying the Rasch Model hands-on introductory workshop, Melbourne, Australia (T. Bond, B&FSteps), Announcement
Dec. 6-8, 2017, Wed.-Fri. In-person workshop: Introductory Rasch Analysis using RUMM2030, Leeds, UK (M. Horton), Announcement
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com