The Root Mean Square Error of Approximation (RMSEA)

as a supplementary statistic to determine fit to the Rasch model with large sample sizes

Georg Rasch mentioned chi-square statistics as a way of evaluating fit of data to the model (Rasch, 1980, p. 25). Ben Wright's Infit and Outfit mean-square statistics are the chi-square divided by their degrees of freedom. However, large sample sizes have always posed problems for significance tests based on chi-square statistics. The issue is that, the larger the sample, the greater the power, and so ever smaller differences are reported as indicating statistically significant misfit between the data and the model. Thus very large sample sizes can detect miniscule differences, and with such samples there is almost no need to undertake a chi-square test as we know that it will be significant (P. Martin-Löf (1974). Indeed, Georg Rasch himself remarked: "On the whole we should not overlook that since a model is never true, but only more or less adequate, deficiencies are bound to show, given sufficient data" (Rasch, 1980, p. 92).

Smith et al. (1998) show that the critical interval values for a Type I error (rejection of a true hypothesis) associated with these statistics varies with sample size. Experience indicates that, while the value of mean-square tends to increase only slowly with sample size, the critical interval associated with a 5% significance level shrinks considerably as sample size increases. Thus a sample of 50 would have a 5% range for Infit of 0.72-1.28, whereas a sample of 500 would have a 5% range of 0.91-1.09. A sample size of 5000 would have a 5% range of 0.97-1.03 (RMT 17:1 p. 918).

In general, large sample sizes will cause most chi-square-based statistics to almost always report a statistically significant difference between the observed data and model expectations, suggesting misfit, regardless of the true situation.

One potential mechanism for accommodating large sample sizes may be to use the Root Mean Square Error of Approximation (RMSEA, Steiger and Lind, 1980) as a supplementary fit. The RMSEA is widely used in Structural Equation Modeling to provide a mechanism for adjusting for sample size where chi-square statistics are used.

Consequently, we set out to test the potential of the RMSEA to supplement the chi-square fit tests reported for Rasch analyses performed by RUMM2030. This investigation focuses on the "summary fit chi-square" (the item trait interaction statistic). The utility of the RMSEA to supplement the interpretation of the chi square fit in larger samples was assessed, along with determination of the level of RMSEA that is consistent with fit to the Rasch model.

Methods

A number of simulations were undertaken with the RUMMss simulation package (Marais I, Andrich D, 2007). Two polytomous item sets of 10 and 20 items with five response categories were simulated with different degrees of fit to the Rasch model. In addition, a set of dichotomous (30) items were also simulated. Perfect fit (100% of the items with simulated discriminations of 1.0), minor deviations (90% with 1.0, 10% with 3.0) and more serious deviations from model expectations (80% with 1.0, 20% with 3.0) were simulated. Each set of simulations was repeated for 200, 500, 2000, 5000, and 10,000 cases. All other parameters were held constant.

The RMSEA was calculated for each simulation, based upon the summary chi-square interaction statistic reported by RUMM2030. The RMSEA formulae can be shown to be equal to:

RMSEA = √ max( [((χ²/df) - 1)/(N - 1)] , 0)

where χ² is the RUMM2030 chi-square value, df is its degrees of freedom and N is the sample size. Notice that the RMSEA has an expected value of zero when the data fit the model. Overfit of the data to the model, χ²/df < 1, is ignored. For a given χ², RMSEA decreases as sample size, N, increases.

Results

In Tables 1-3, the average RMSEA for each simulated condition is reported. Within each column of each Table, the RMSEA is largely invariant as the sample size increases, as we had hoped.

Across each row of each Table, for sample sizes of 500 or more, the RMSEA is sensitive to increasing misfit. Thus it may be appropriate to use this supplementary fit statistic in the presence of sample sizes of 500 or more cases, to inform if sample size is inflating the chi-square statistic, and hence its significance.

Conclusion

The results of this study suggest that investigations of fit to the Rasch model using RUMM2030 and specifically the item-trait interaction chi-square fit statistic, in the presence of large sample sizes, can be supplemented through applying the RMSEA statistic. RMSEA values of < 0.02 with sample sizes of 500+, and certainly 1000+, may indicate that the data do not underfit the model, and that the chi-square was inflated by sample size.

Alan Tennant, Department of Rehabilitation Medicine, Faculty of Medicine and Health, The University of Leeds, UK
Julie F. Pallant, Rural Health Academic Centre, University of Melbourne, Australia.

References

Marais I, Andrich D (2007)\: RUMMss. Rasch Unidimensional Measurement Models Simulation Studies Software. The University of Western Australia, Perth.

Martin-Löf P. (1974). The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and observational data. Scandinavian Journal of Statistics, 1:3.

Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press.

Smith, R. M, Schumacker RE, Bush MJ. (1998). Using item mean squares to evaluate fit to the Rasch model. Journal of Outcome Measurement, 2: 66-78.

Steiger, J. H. and Lind, J. (1980) Statistically-based tests for the number of common factors. Paper presented at the Annual Spring Meeting of the Psychometric Society, Iowa City.

The Root Mean Square Error of Approximation (RMSEA) as a supplementary statistic to determine fit to the Rasch model with large sample sizes. Alan Tennant & Julie F. Pallant ... Rasch Measurement Transactions, 2012, 25:4, 1348-9

Rasch Books and Publications
Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, 2nd Edn. George Engelhard, Jr. & Jue Wang	Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene	Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver	Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone	Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes	Statistical Analyses for Language Testers (Facets), Rita Green	Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind	Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M	Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind	Rasch Measurement: Applications, Khine	Winsteps Tutorials - free Facets Tutorials - free	Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre	Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan
Other Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free	An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse	Rasch Measurement Theory Analysis in R, Wind, Hua	Applying the Rasch Model in Social Sciences Using R, Lamprianou	El modelo métrico de Rasch: Fundamentación, implementación e interpretación de la medida en ciencias sociales (Spanish Edition), Manuel González-Montesinos M.
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar	Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch	Rasch Models for Measurement, David Andrich	Constructing Measures, Mark Wilson	Best Test Design - free, Wright & Stone Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias	Diseño de Mejores Pruebas - free, Spanish Best Test Design	A Course in Rasch Measurement Theory, Andrich, Marais	Rasch Models in Health, Christensen, Kreiner, Mesba	Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen

Forum

Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

The URL of this page is www.rasch.org/rmt/rmt254d.htm

Website: www.rasch.org/rmt/contents.htm