Data Analysis and Fit

Effective data analysis begins with the realization that neither the model nor the data are true. Both are inventions and fictions. The purpose of a model is (1) to represent important requirements in a decisive way (the more decisive the better) and (2) to be useful for the construction of knowledge. Throughout physics, chemistry, business and sports, a central requirement for a measurement model is that, as far as the model is concerned, comparisons of objects are independent of which agents are used, and comparisons of agents are independent of which objects are used. The model which operates in this manner to construct quantities from ordered qualities is the Rasch model.

The Rasch model is a conceptual ideal. No data can ever fit this (or any other) model. But the degree to which the data do cooperate with the model, and the local departures of the data from the ideal of the model, guide the management of the data. If at all possible, neither the data nor the model is accepted or rejected totally. Since the model represents our requirements, to reject it is to reject our line of inquiry. Since the data set contains what the world has to tell us, to reject it is to state that the world, or, at least that part of it, has no information relating to our concern. Very rarely, a variable is so poorly conceptualized, or a data set so confused, that no meaningful analysis can be performed. Then it is the data set, not the model, that is rejected.

The measurement model provides local indicators, fit statistics, of the degree to which the data is cooperating with the model's requirements. These local indicators identify pieces of data consistent with the main pattern, and also those least consistent. Those data consistent with the main pattern, usually the vast majority for any data set collected with any degree of thought and care, become the basis for measures and the construction of knowledge. The inconsistent data become a source of further discovery. They need to be studied in more detail, and, when the inconsistency is due to irrelevant activity, those pieces of data need to be put to one side or manipulated to be in accord with the main pattern.

When diagnosis of inconsistent data indicates that an examinee has a spoiled performance, (perhaps there is clear evidence of guessing, carelessness or such like,) the examinee's measure is also spoiled. To draw inferences from this measure about the ability of the examinee is to risk being misinformed. Sometimes the measure can be rescued by putting to one side the spoiled part of the data, and using only the part that cooperates with the model. Sometimes there is nothing to rescue. The data provides no firm base from which to infer the examinee's ability. Of course, if the purpose of the analysis is merely to award a prize for the most "correctly" marked squares on an answer sheet, even coffee stains, smudges and doodles count! But that is not science.

When an item has not functioned in the same way as most other items, it is not acting as an agent assisting in the measurement of the same variable. It is acting like a broken ruler. It is contributing not to measurement, but to noise, until either it is removed from the analysis, or, preferably mended. A common form of repair is to correct an errant scoring key.

What is the precise role of fit statistics in identifying spoiled performances or broken items? Is accurate knowledge of the distributions of the fit statistics for unspoiled performances, the "null" distributions, necessary for data analysis to proceed fairly, rationally, systematically, routinely and automatically? Automatically?? Empirical data are too complex in their origin for any automatic decision maker to be entirely successful. Fit statistics provide guidance, but not decisions. Decisions must be made within a larger context and for richer reasons than the null distribution of a fit statistic can provide. A substantive explanation of data identified as irregular is more important than the absolute magnitude of significance level of some fit statistic, chosen purely for its ability to aid the hunt for inconsistency.

Even were the null distribution available, it too is an ideal from which any real set of data inevitably falls short. Therefore, it is sufficient for thoughtful data analysis to know that the fit statistics can be relied upon (1) to order the persons from most underfitting to most overfitting, (2) to center this ordering near some easily remembered numerical value, such as 0 or 1, as a guide to where the transition is from under to overfitting and (3) to provide some frame of reference, so that a fit statistic of value, say 2, continues to represent about the same degree of spoiling for similar data collected under similar conditions. Whatever the values of the fit statistics, extreme cases require examination to discover why they are spoiled, or to confirm their freedom from spoilage.

Ben Wright

Data analysis and fit. Wright BD. Rasch Measurement Transactions 1994 7:4 p.324

Data analysis and fit. Wright BD. … Rasch Measurement Transactions, 1994, 7:4 p.324

Rasch Books and Publications
Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, 2nd Edn. George Engelhard, Jr. & Jue Wang	Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene	Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver	Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone	Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes	Statistical Analyses for Language Testers (Facets), Rita Green	Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind	Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M	Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind	Rasch Measurement: Applications, Khine	Winsteps Tutorials - free Facets Tutorials - free	Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre	Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan
Other Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free	An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse	Rasch Measurement Theory Analysis in R, Wind, Hua	Applying the Rasch Model in Social Sciences Using R, Lamprianou	El modelo métrico de Rasch: Fundamentación, implementación e interpretación de la medida en ciencias sociales (Spanish Edition), Manuel González-Montesinos M.
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar	Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch	Rasch Models for Measurement, David Andrich	Constructing Measures, Mark Wilson	Best Test Design - free, Wright & Stone Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias	Diseño de Mejores Pruebas - free, Spanish Best Test Design	A Course in Rasch Measurement Theory, Andrich, Marais	Rasch Models in Health, Christensen, Kreiner, Mesba	Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen

Forum

Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

The URL of this page is www.rasch.org/rmt/rmt74f.htm

Website: www.rasch.org/rmt/contents.htm