Toward Practical Significance of Rasch Scores in International Studies in Education:
More than Statistical Significance and Effect Size

Both in social sciences and in education, the scales that are used can easily and accurately lead to statistical significance, calculation of effect size indices, or computation of the confidence interval . However, unfortunately, practical significance of these scores can scarcely be obtained. For example, what does it mean to a teacher to be two points happier with his job than another one on a Likert scale? Almost worse: What does it mean to a school to be compared to the average national results in mathematics? Just to say that there is a statistical significant difference between the school score and the national average scores is not very informative for the school.

Statisticians proposed a compromise to these kinds of situations. For this purpose, they suggest using effect size indices and confidence intervals in replacement of significance tests. They are interesting devices, though not as much when it comes to the practical significance of scores and to the difference between these scores.

In the context of comparisons made with the results of international and national surveys in education, like the TEIMS, PIRLS and PISA, practical significance of these scores would lead researchers to more useful interpretations. It would lead to more meaningful comparisons of countries and states. Important decisions would be easier to make.

In the context of these international surveys, researchers tried to find a way to lead to practical significance of the obtained mathematics, science, or reading literacy scores. For example, Cartwright, Lalancette, Mussio, and Xing (2003) related these international scores to results at a smaller provincial scale in Canada. They found that the provincial results in reading of British Columbia can be related with a good precision to the international PISA reading comprehension literacy score. While these findings don't permit a direct practical interpretation, they tell us, at least, that it is possible to link results obtained at one survey, the PISA, to those obtained at another one, the British-Columbia.

Figure 1. Relation between Rasch and TEIMS 2003 scores, schooling days and number of books at home

More interesting, and more practical, is Willms' proposition (2004a, 2004b). Willms tries to translate the international Rasch scores results from PISA into a metric of school years and school days. This is an attractive idea, because it leads directly to a metric that allows administrative decision and teaching effect interpretation. Thus, it would be legitimate to say that to reach an augmentation of one point on one of PISA literacy scales, a fixed number of schooling days would be necessary. Willms, in preliminary works, finds out that a difference of one point on the PISA scale is equivalent to about 5 schooling days. He didn't report on which literacy scale the equivalence was computed. He did this estimation by considering the fact that, because of the annual inscription date in each state and country, the 15-years-old students participating in the survey can be on two different school levels. This fact permits him to compute the effect of one schooling year on the PISA scores. The average schooling year in the 12 countries, where he was able to obtain the information about the level and the date of inscription, was equal to 172 days. Wherefore he observed that a schooling year correspond to a difference score of 34.30, he extrapolates that a one point difference is equivalent to about 5 schooling days (172/34.30).

By themselves, Willms' findings are of interest, being now able to give a clear interpretation of differences in literacy scores. However, if we also consider Cartwright et al., we can think that the equivalence formula obtained by Willms can be related to the one of other international and national educational surveys. Consider that the PISA scores are on a scale with a mean of 500 and a standard deviation of 100 and that, for example, the TEIMS Rasch scale has a mean of 150 and a standard deviation of 10. Thus, only using the standard deviation ratio, a one point difference at the TEIMS scale would be equivalent to a 10 points difference at the PISA scale: sPISA / sTIMSS = 100/10. An illustration of the relation between TIMSS 2003 scores, schooling days, and number of books at home is presented in Figure 1. It can be seen that a mean difference of 32 books at home (37 - 5) corresponds approximately to one school year (178.50 days).

A note of caution is to be considered. More research work is still to be done on this topic and it will have to consider the specific literacy scale, how old the students are, as well as their school level. More important, however, we think that the equivalence between the schooling days and the difference in the literacy scores would have to be computed independently for each country, not by an average on the 12 countries used by Willms which are so different in schooling practice.

Gilles Raîche, Université du Québec à Montréal
Claire IsaBelle, Université d'Ottawa
Martin Riopel, Université du Québec à Montréal
Cartwright, F., Lalancette, D., Mussio, J. and Xing, D. (2003). Linking provincial student assessments with national and international assessments. Report no 81-595-MIE2003005. Ottawa, Canada: Statistics Canada.
Willms, J. D. (2004a). Reading achievement in Canada and the United States : Findings from the OECD programme for international student assessment. Final Report no SP-601-05-04E. Ottawa, Canada: Human Resources and Skills Development Canada.
Willms, J. D. (2004b). Variation in literacy skills among Canadian provinces: Findings from the OECD PISA. Report no 81-595-MIE2004012. Ottawa, Canada: Statistics Canada.