A Method of Estimating the Item Parameter from Time on Task

The purpose of this essay is to revisit Chapter 3 of Probabilistic Models (Rasch, 1960) and to consider to what extent the arguments from this chapter might be applied to individual items in a computer based math test. The intention is to look for inspiration from Rasch, but not to follow his methodology exactly.

Hippisley (1999) showed that total completion times from a computer based math test conformed to the (Rasch 1960) reading rate model. However, the assumption of item homogeneity and uniform student speed, rather limits the usefulness of that analysis. In reality, test items are not homogenous, and students do not course through a test at a uniform speed.

Theory

Rasch considered the "speededness" of students reading a text in two ways. For those students who did not complete the text within a prescribed time limit, he treated the number of words read within the time as the stochastic variable. For the students who did complete the test (within the time limit), he treated the time actually taken as the stochastic variable.

For the former group, if λ is mean or expected reading rate, the probability of reading a words in time t is given as:

The probability that no words are read in time t is a special case, where N = 0. This reduces down to:

This expression may be applied to a student tackling a single item in a math test. And while it derives from Expression (1) for reading rates, and incorporates λ, (which could be interpreted as an expected item completion rate), the application will be from measuring the time during which the item is not completed.

Rasch (1960) broke down λ into two factors. He argued that the ratio of the reading rates between two pupils A and B is interesting if it applies to a number of reading texts. So if in a series of texts, 1, 2, ..., i, pupil A reads twice as fast as B:

and

Dividing:

Generalising:

So the ratio of the mean or expected reading rates of two texts is independent of the pupils. That ratio tells is something about the texts (relative ease of reading or some other applicable descriptor) and it might be given the parameter ε:

Rearranging:

Rasch (1960) described the term λv1, the reading speed of pupil v in a base text, as the person parameter ζ_v. He also defined the reciprocal of ε as the difficulty δ of a text.

When applying this argument to an individual item in a computer based math test, it should be born in mind that the focus is on a single event. It is impossible to predict exactly when the event will occur, and it is equally impossible to estimate the value of λ from knowing when the event occurred. To overcome this conundrum, a method might be borrowed from natural science.

When physicists consider a sample of radioactive material comprising many atoms, they apply the Law of Large Numbers (Khoshnevisan, 2007), which essentially states that if you run an experiment N times, where N is a very large number, if p is the probability of an event, the number of times the event actually occurs will approximate to Np. From Expression (2) above, the probability of a word not being read, or a math item not being completed, or a radioactive atom not decaying, in time t, is e^-λt. Studying a sample originally comprising N₀ atoms, is like running an experiment N₀ times. After time t, the approximate number N of atoms, which have not decayed, will be given by:

The time t_h taken for N to become exactly half N₀ is known as the half-life of the material.

So if you had a room full of clones all addressing the same item at the same time, you could estimate λ from the time it takes half of them to complete the item. Clones are not easy to come by, but there is another formula from physics which deals with composite radioactive material (L'Annunziata, 2012), and which could be applied to a heterogeneous set of pupils.

In the case of two elements, if the decay rate of Element 1 with N₁ atoms is λ₁ and that of Element 2 with N₂ atoms is λ₂, the combined decay rate λ_c, or number of atoms decaying per unit of time is:

In psychometrics, there is usually assumed to be just one pupil of each type, so for two pupils, the combined item completion rate becomes:

Reverting to the Rasch notation of Expression (5), if these pupils are addressing item i:

If the same pupils address a second item j:

Dividing:

So the ratio of the combined expected completion rates becomes the ratio of the easiness of the two items, and is independent of the person parameters of the two pupils. A similar argument applies to three or more pupils. Furthermore, the combined expected completion rate can be estimated for each item from the median completion time on each item using Expression (7).

Illustration

Figures 1 and 2 show for 85 West Australian primary school students, all of whom completed (correctly) the items "4+4", "3+5", and "12+8" in a computer-based math test, the completion times on item "3+5" against those on item "4+4", and the completion times on item "12+8" against those on item "4+4". This triple intersection set arose from a universal set of 14,480 student-item interactions. The settings were informal, with class teachers using the computer-based test as a regular class activity, as opposed to a formal exam.

Completion times in secs for item 3+5 against 4+4 Figure 1.

Completion times in secs for item 12+8 against 4+4 Figure 2.

Table 1 shows the median completion times in seconds and an estimate of λ_c for each item. The table also shows ratios of easiness ε and difficulty δ. From the table, if item "4+4" is treated as the base item, item "3+5" seems to be approximately two thirds as easy or 1.5 times as difficult, while "12+8" seems to be approximately half as easy or twice as difficult.

The purpose of this essay was to set out a method of estimating the Rasch item parameter from time on task. A method has been laid out, and an illustration has been given. The illustration looked at just three items, all of which had been addressed by the same pupils. Extending the method to cover all of the possible items, which might come up in a simple math test, will either require a very large sample of student-item interactions, or the development of a system, which does not require exactly the same pupils to address every item.

Jonathan Hippisley
Email: jhipp -/at\- softway.org

References

Hippisley J (1999) Looking at Data from an Interactive Arithmetic Test from the Perspective of a Probabilistic Model. Education Research & Perspectives, (25)2, 59-67.

Khoshnevisan D (2007) Probability Graduate Studies in Mathematics, 80. American Mathematical Society.

L'Annunziata M (2012) Handbook of Radioactivity Analysis. Academic Press.

Rasch G (1960) Probabilistic models for some intelligence and attainment tests University of Chicago Press, Chicago

A Method of Estimating the Item Parameter from Time on Task. Jonathan Hippisley … Rasch Measurement Transactions, 2012, 26:3 p. 1380-2

Rasch Books and Publications
Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, 2nd Edn. George Engelhard, Jr. & Jue Wang	Applying the Rasch Model (Winsteps, Facets) 4th Ed., Bond, Yan, Heene	Advances in Rasch Analyses in the Human Sciences (Winsteps, Facets) 1st Ed., Boone, Staver	Advances in Applications of Rasch Measurement in Science Education, X. Liu & W. J. Boone	Rasch Analysis in the Human Sciences (Winsteps) Boone, Staver, Yale
Introduction to Many-Facet Rasch Measurement (Facets), Thomas Eckes	Statistical Analyses for Language Testers (Facets), Rita Green	Invariant Measurement with Raters and Rating Scales: Rasch Models for Rater-Mediated Assessments (Facets), George Engelhard, Jr. & Stefanie Wind	Aplicação do Modelo de Rasch (Português), de Bond, Trevor G., Fox, Christine M	Appliquer le modèle de Rasch: Défis et pistes de solution (Winsteps) E. Dionne, S. Béland
Exploring Rating Scale Functioning for Survey Research (R, Facets), Stefanie Wind	Rasch Measurement: Applications, Khine	Winsteps Tutorials - free Facets Tutorials - free	Many-Facet Rasch Measurement (Facets) - free, J.M. Linacre	Fairness, Justice and Language Assessment (Winsteps, Facets), McNamara, Knoch, Fan
Other Rasch-Related Resources: Rasch Measurement YouTube Channel
Rasch Measurement Transactions & Rasch Measurement research papers - free	An Introduction to the Rasch Model with Examples in R (eRm, etc.), Debelak, Strobl, Zeigenfuse	Rasch Measurement Theory Analysis in R, Wind, Hua	Applying the Rasch Model in Social Sciences Using R, Lamprianou	El modelo métrico de Rasch: Fundamentación, implementación e interpretación de la medida en ciencias sociales (Spanish Edition), Manuel González-Montesinos M.
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar	Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch	Rasch Models for Measurement, David Andrich	Constructing Measures, Mark Wilson	Best Test Design - free, Wright & Stone Rating Scale Analysis - free, Wright & Masters
Virtual Standard Setting: Setting Cut Scores, Charalambos Kollias	Diseño de Mejores Pruebas - free, Spanish Best Test Design	A Course in Rasch Measurement Theory, Andrich, Marais	Rasch Models in Health, Christensen, Kreiner, Mesba	Multivariate and Mixture Distribution Rasch Models, von Davier, Carstensen

Forum

Rasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

The URL of this page is www.rasch.org/rmt/rmt263d.htm,

Website: www.rasch.org/rmt/contents.htm,