The Root Mean Square Error of Approximation (RMSEA)

as a supplementary statistic to determine fit to the Rasch model with large sample sizes

Table 1. RMSEA Results for Set 1
(10 polytomous items)
Sample SizeNo Misfit10% Misfit20% Misfit
2000.0000.0000.033
5000.0040.0240.035
20000.0110.0240.030
50000.0140.0240.031
100000.0140.0240.031
 
Table 2. RMSEA Results for Set 2
(20 polytomous items)
Sample SizeNo Misfit10% Misfit20% Misfit
2000.0000.0530.043
5000.0000.0240.040
20000.0040.0310.038
50000.0060.0300.038
100000.0090.0310.038
 
Table 3. RMSEA Results for Set 3
(20 dichotomous items)
Sample SizeNo Misfit10% Misfit20% Misfit
2000.0000.0610.073
5000.0160.0190.035
20000.0130.0260.040
50000.0110.0270.040
100000.0120.0270.041

Georg Rasch mentioned chi-square statistics as a way of evaluating fit of data to the model (Rasch, 1980, p. 25). Ben Wright's Infit and Outfit mean-square statistics are the chi-square divided by their degrees of freedom. However, large sample sizes have always posed problems for significance tests based on chi-square statistics. The issue is that, the larger the sample, the greater the power, and so ever smaller differences are reported as indicating statistically significant misfit between the data and the model. Thus very large sample sizes can detect miniscule differences, and with such samples there is almost no need to undertake a chi-square test as we know that it will be significant (P. Martin-Löf (1974). Indeed, Georg Rasch himself remarked: "On the whole we should not overlook that since a model is never true, but only more or less adequate, deficiencies are bound to show, given sufficient data" (Rasch, 1980, p. 92).

Smith et al. (1998) show that the critical interval values for a Type I error (rejection of a true hypothesis) associated with these statistics varies with sample size. Experience indicates that, while the value of mean-square tends to increase only slowly with sample size, the critical interval associated with a 5% significance level shrinks considerably as sample size increases. Thus a sample of 50 would have a 5% range for Infit of 0.72-1.28, whereas a sample of 500 would have a 5% range of 0.91-1.09. A sample size of 5000 would have a 5% range of 0.97-1.03 (RMT 17:1 p. 918).

In general, large sample sizes will cause most chi-square-based statistics to almost always report a statistically significant difference between the observed data and model expectations, suggesting misfit, regardless of the true situation.

One potential mechanism for accommodating large sample sizes may be to use the Root Mean Square Error of Approximation (RMSEA, Steiger and Lind, 1980) as a supplementary fit. The RMSEA is widely used in Structural Equation Modeling to provide a mechanism for adjusting for sample size where chi-square statistics are used.

Consequently, we set out to test the potential of the RMSEA to supplement the chi-square fit tests reported for Rasch analyses performed by RUMM2030. This investigation focuses on the "summary fit chi-square" (the item trait interaction statistic). The utility of the RMSEA to supplement the interpretation of the chi square fit in larger samples was assessed, along with determination of the level of RMSEA that is consistent with fit to the Rasch model.

Methods

A number of simulations were undertaken with the RUMMss simulation package (Marais I, Andrich D, 2007). Two polytomous item sets of 10 and 20 items with five response categories were simulated with different degrees of fit to the Rasch model. In addition, a set of dichotomous (30) items were also simulated. Perfect fit (100% of the items with simulated discriminations of 1.0), minor deviations (90% with 1.0, 10% with 3.0) and more serious deviations from model expectations (80% with 1.0, 20% with 3.0) were simulated. Each set of simulations was repeated for 200, 500, 2000, 5000, and 10,000 cases. All other parameters were held constant.

The RMSEA was calculated for each simulation, based upon the summary chi-square interaction statistic reported by RUMM2030. The RMSEA formulae can be shown to be equal to:

RMSEA = √ max( [((χ²/df) - 1)/(N - 1)] , 0)

where χ² is the RUMM2030 chi-square value, df is its degrees of freedom and N is the sample size. Notice that the RMSEA has an expected value of zero when the data fit the model. Overfit of the data to the model, χ²/df < 1, is ignored. For a given χ², RMSEA decreases as sample size, N, increases.

Results

In Tables 1-3, the average RMSEA for each simulated condition is reported. Within each column of each Table, the RMSEA is largely invariant as the sample size increases, as we had hoped.

Across each row of each Table, for sample sizes of 500 or more, the RMSEA is sensitive to increasing misfit. Thus it may be appropriate to use this supplementary fit statistic in the presence of sample sizes of 500 or more cases, to inform if sample size is inflating the chi-square statistic, and hence its significance.

Conclusion

The results of this study suggest that investigations of fit to the Rasch model using RUMM2030 and specifically the item-trait interaction chi-square fit statistic, in the presence of large sample sizes, can be supplemented through applying the RMSEA statistic. RMSEA values of < 0.02 with sample sizes of 500+, and certainly 1000+, may indicate that the data do not underfit the model, and that the chi-square was inflated by sample size.

Alan Tennant, Department of Rehabilitation Medicine, Faculty of Medicine and Health, The University of Leeds, UK
Julie F. Pallant, Rural Health Academic Centre, University of Melbourne, Australia.

References

Marais I, Andrich D (2007)\: RUMMss. Rasch Unidimensional Measurement Models Simulation Studies Software. The University of Western Australia, Perth.

Martin-Löf P. (1974). The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and observational data. Scandinavian Journal of Statistics, 1:3.

Rasch, G. (1980). Probabilistic models for some intelligence and attainment tests. Chicago: University of Chicago Press.

Smith, R. M, Schumacker RE, Bush MJ. (1998). Using item mean squares to evaluate fit to the Rasch model. Journal of Outcome Measurement, 2: 66-78.

Steiger, J. H. and Lind, J. (1980) Statistically-based tests for the number of common factors. Paper presented at the Annual Spring Meeting of the Psychometric Society, Iowa City.



The Root Mean Square Error of Approximation (RMSEA) as a supplementary statistic to determine fit to the Rasch model with large sample sizes. Alan Tennant & Julie F. Pallant ... Rasch Measurement Transactions, 2012, 25:4, 1348-9




Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
May 17 - June 21, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 12 - 14, 2024, Wed.-Fri. 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024
June 21 - July 19, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 5 - Aug. 6, 2024, Fri.-Fri. 2024 Inaugural Conference of the Society for the Study of Measurement (Berkeley, CA), Call for Proposals
Aug. 9 - Sept. 6, 2024, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 4 - Nov. 8, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 17 - Feb. 21, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 16 - June 20, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 20 - July 18, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com
Oct. 3 - Nov. 7, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

The URL of this page is www.rasch.org/rmt/rmt254d.htm

Website: www.rasch.org/rmt/contents.htm