Reasonable mean-square fit values.

"Less control over the data → more off-dimensional behavior → more departures in the data from the Rasch model"

Though the ideal for measurement construction is that data fit the Rasch model, all empirical data departs from the model to some extent. But how much unmodelled noise is tolerable? Conventional statisticians base their decisions on significance tests, but these are heavily influenced by sample size.

"Very large samples form a special source of problems. This is because no model can ever be supposed to be perfectly fitted by data, so with a sufficiently large sample any model would have to be discarded. In connection with the problem Martin-Löf (1974) stated: 'This indicates that for large sets of data it is too destructive to let an ordinary significance test decide whether or not to accept a proposed statistical model [or data], because, with few exceptions, we know that we shall have to reject it even without looking at the data simply because the number of observations is so large. In such cases, we need instead a quantitative measure of the size of the discrepancy between the statistical model and the observed set of data.'" (Gustafson 1980).

Convenient quantitative measures of fit discrepancy are mean-square residual summary statistics, such as OUTFIT and INFIT. These statistics have expectation 1.0, and range from 0 to infinity. Mean-squares greater than 1.0 indicate underfit to the Rasch model, i.e., the data are less predictable than the model expects. Mean-squares less than 1.0 indicate overfit to the Rasch model, i.e., the data are more predictable than the model expects. A mean-square of 1.2 indicates that there is 20% more randomness (i.e., noise) in the data than modelled. A mean-square of 0.7 indicates a 30% deficiency in Rasch-model-predicted randomness (i.e., the data are too Guttman-like), which implies 100*(1-0.7)/0.7 = 43% more ambiguity in the inferred measure than modelled (e.g., the item difficulty estimated from low-ability persons differs noticeably from the item difficulty estimated from high-ability persons).

Reasonable Item Mean-square Ranges for INFIT and OUTFIT
Type of Test	Range
MCQ (High stakes) MCQ (Run of the mill) Rating scale (survey) Clinical observation Judged (agreement encouraged)	0.8 - 1.2 0.7 - 1.3 0.6 - 1.4 0.5 - 1.7 0.4 - 1.2

When is a mean-square too large or too small? There are no hard-and-fast rules. Particular features of a testing situation, e.g., mixing item types or off-target testing, can produce idiosyncratic mean-square distributions. Nevertheless, here, as a rule of thumb, are some reasonable ranges for item mean-square fit statistics. In Classical Test Theory, 3-PL IRT, and conventional statistics, low mean-squares are considered good. In Rasch theory, they indicate some redundancy in the responses, but they do no harm.

Ben Wright & John Michael Linacre

David Andrich (2024, May 30) in email to the Matilda Bay Club: "One way is to delete the least discriminating item, as seen ... by the fit statistics [largest mean-square], and see if the Rasch reliability, or Person Separation, drops. If it does, then it means the item had added more information than noise, and could be a reason for keeping it in. On the other hand if the reliability increases, then the item was adding more noise than in information, and from a statistical perspective, could be dropped. But you would want to go back and understand why an item that someone thought worked consistently with the other items did not do so."

J.-E. Gustafson (1980) Testing and obtaining fit of data to the Rasch model. British Journal of mathematical and Statistical Psychology, 33, p.220.

P. Martin-Löf (1974) The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and observational data. Scandinavian Journal of Statistics, 1, 3.

Note by Linacre: Informal simulations studies and experience analyzing hundreds of datasets indicate that:

The Mean-Square statistic is also called the Relative Chi-Square and the Normed Chi-Square. According to www.psych-it.com.au/Psychlopedia/article.asp?id=277, in Structural Equation Modeling, the criterion for acceptance varies across researchers, ranging from less than 2 (Ullman, 2001) to less than 5 (Schumacker & Lomax, 2004).

[Later:] Overfit (mean-squares less than 1.0).

• Mean-squares are forced to average near 1.0. So, if there are very large mean-squares (for instance, due to random lucky guessing), this forces there also to be low mean-squares.
  Advice: always investigate and remove items with high mean-squares before looking at items with low mean-squares.
• Mean-squares less than 1.0 are too predictable. They do not contradict what we know, but they do not tell us much that is new about what we want to know. So overfitting items are inefficient.
  Advice:
  i) if we are developing a new test: replace overfitting items with more efficient items.
  ii) if we are analyzing data from an existing test: take no action, even an item with a very low mean-square tells us a little something that is new and useful. We don't want to waste any useful information.
• To reduce a large set of items to a small Rasch-compliant set:
  
  1. Eliminate the underfitting items >1.2 - this removes the uncooperative items
  2. Reanalyze the data. You will notice that the mean-square fit statistics recentralize.
  3. Now eliminate the underfitting and overfitting items (>1.2 and <0.8) - this optimizes the selection of the reasonably behaved items.

See also "Size vs. Significance: Mean-Square and Standardized Chi-Square Fit Statistics" www.rasch.org/rmt/rmt171n.htm

Item Fit and Person Fit

From a statistical Rasch perspective, persons and items are exactly the same. They are merely parameters of the Rasch model. So the fit criteria would be exactly the same. But, from a substantive perspective, persons and items differ. We expect the items to be better-behaved than the persons. We also expect item difficulties to continue into the future, but we expect person abilities to change. Also, we expect items to be encountered by many, many persons, but persons to encounter relatively few items. Consequently, we are usually stricter in our application of fit rules to items than to persons. A few maverick persons in a dataset don't worry us - they will have negligible impact on anything else. But a few maverick items raise questions about test administration, data entry accuracy, the definition of the latent variable, etc. We will immediately focus our attention on them because they may be symptomatic of a more pervasive problem, such as the wrong key for a multiple-choice test, or reversed-coded items on a survey.

Reasonable mean-square fit values. Wright BD, Linacre JM … Rasch Measurement Transactions, 1994, 8:3 p.370

Rasch Books and Publications
Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, 2nd Edn. George Engelhard, Jr. & Jue Wang	Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene	Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver	Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone	Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes	Statistical Analyses for Language Testers (Facets), Rita Green	Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind	Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M	Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind	Rasch Measurement: Applications, Khine	Winsteps Tutorials - free Facets Tutorials - free	Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre	Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan
Other Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free	An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse	Rasch Measurement Theory Analysis in R, Wind, Hua	Applying the Rasch Model in Social Sciences Using R, Lamprianou	El modelo métrico de Rasch: Fundamentación, implementación e interpretación de la medida en ciencias sociales (Spanish Edition), Manuel González-Montesinos M.
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar	Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch	Rasch Models for Measurement, David Andrich	Constructing Measures, Mark Wilson	Best Test Design - free, Wright & Stone Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias	Diseño de Mejores Pruebas - free, Spanish Best Test Design	A Course in Rasch Measurement Theory, Andrich, Marais	Rasch Models in Health, Christensen, Kreiner, Mesba	Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen

Forum

Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

The URL of this page is www.rasch.org/rmt/rmt83b.htm

Website: www.rasch.org/rmt/contents.htm