A Method of Estimating the Item Parameter from Time on Task

The purpose of this essay is to revisit Chapter 3 of Probabilistic Models (Rasch, 1960) and to consider to what extent the arguments from this chapter might be applied to individual items in a computer based math test. The intention is to look for inspiration from Rasch, but not to follow his methodology exactly.

Hippisley (1999) showed that total completion times from a computer based math test conformed to the (Rasch 1960) reading rate model. However, the assumption of item homogeneity and uniform student speed, rather limits the usefulness of that analysis. In reality, test items are not homogenous, and students do not course through a test at a uniform speed.

Theory

Rasch considered the "speededness" of students reading a text in two ways. For those students who did not complete the text within a prescribed time limit, he treated the number of words read within the time as the stochastic variable. For the students who did complete the test (within the time limit), he treated the time actually taken as the stochastic variable.

For the former group, if λ is mean or expected reading rate, the probability of reading a words in time t is given as:

  P{a|t} = ((λt)a/a!)e-λt (1)

The probability that no words are read in time t is a special case, where N = 0. This reduces down to:

  P{0 | t} = e-λt (2)

This expression may be applied to a student tackling a single item in a math test. And while it derives from Expression (1) for reading rates, and incorporates λ, (which could be interpreted as an expected item completion rate), the application will be from measuring the time during which the item is not completed.

Rasch (1960) broke down λ into two factors. He argued that the ratio of the reading rates between two pupils A and B is interesting if it applies to a number of reading texts. So if in a series of texts, 1, 2, ..., i, pupil A reads twice as fast as B:

  λA1 = 2λB1  

and

  λAi = 2λBi  

Dividing:

  λAiA1 = λBiB1  

Generalising:

  λli11 = λNiN1 (3)

So the ratio of the mean or expected reading rates of two texts is independent of the pupils. That ratio tells is something about the texts (relative ease of reading or some other applicable descriptor) and it might be given the parameter ε:

  λviv1 = ε  

Rearranging:

  λvi = λv1ε (4)

Rasch (1960) described the term λv1, the reading speed of pupil v in a base text, as the person parameter ζv. He also defined the reciprocal of ε as the difficulty δ of a text.

When applying this argument to an individual item in a computer based math test, it should be born in mind that the focus is on a single event. It is impossible to predict exactly when the event will occur, and it is equally impossible to estimate the value of λ from knowing when the event occurred. To overcome this conundrum, a method might be borrowed from natural science.

When physicists consider a sample of radioactive material comprising many atoms, they apply the Law of Large Numbers (Khoshnevisan, 2007), which essentially states that if you run an experiment N times, where N is a very large number, if p is the probability of an event, the number of times the event actually occurs will approximate to Np. From Expression (2) above, the probability of a word not being read, or a math item not being completed, or a radioactive atom not decaying, in time t, is e-λt. Studying a sample originally comprising N0 atoms, is like running an experiment N0 times. After time t, the approximate number N of atoms, which have not decayed, will be given by:

  N(t) = N0e-λt (5)

The time th taken for N to become exactly half N0 is known as the half-life of the material.

  N0/2 = N0e-λth  
  2 = eλth  
  ln(2) = λth  
  λ = ln(2)/th (6)

So if you had a room full of clones all addressing the same item at the same time, you could estimate λ from the time it takes half of them to complete the item. Clones are not easy to come by, but there is another formula from physics which deals with composite radioactive material (L'Annunziata, 2012), and which could be applied to a heterogeneous set of pupils.

In the case of two elements, if the decay rate of Element 1 with N1 atoms is λ1 and that of Element 2 with N2 atoms is λ2, the combined decay rate λc, or number of atoms decaying per unit of time is:

  -dN/dt = N1λ1 + N2λ2  

In psychometrics, there is usually assumed to be just one pupil of each type, so for two pupils, the combined item completion rate becomes:

  λc = λ1 + λ2  

Reverting to the Rasch notation of Expression (5), if these pupils are addressing item i:

  λci = ζ1εi + ζ2εi  
  λci = εi1 + ζ2εi)  

If the same pupils address a second item j:

  λcj = εj1 + ζ2εi)  

Dividing:

  λcicj = εi1 + ζ2εi) / εj1 + ζ2εi)  
  λcicj = εi / εj (7)

So the ratio of the combined expected completion rates becomes the ratio of the easiness of the two items, and is independent of the person parameters of the two pupils. A similar argument applies to three or more pupils. Furthermore, the combined expected completion rate can be estimated for each item from the median completion time on each item using Expression (7).

Illustration

Figures 1 and 2 show for 85 West Australian primary school students, all of whom completed (correctly) the items "4+4", "3+5", and "12+8" in a computer-based math test, the completion times on item "3+5" against those on item "4+4", and the completion times on item "12+8" against those on item "4+4". This triple intersection set arose from a universal set of 14,480 student-item interactions. The settings were informal, with class teachers using the computer-based test as a regular class activity, as opposed to a formal exam.


Completion times in secs
for item 3+5 against 4+4

Figure 1.
Completion times in secs
for item 12+8 against 4+4

Figure 2.

Table 1
ItemMedian Time (s)λε ratioδ ratio
4+41.580.441.001.00
3+52.500.280.631.58
12+85.000.140.502.00

Table 1 shows the median completion times in seconds and an estimate of λc for each item. The table also shows ratios of easiness ε and difficulty δ. From the table, if item "4+4" is treated as the base item, item "3+5" seems to be approximately two thirds as easy or 1 times as difficult, while "12+8" seems to be approximately half as easy or twice as difficult.

The purpose of this essay was to set out a method of estimating the Rasch item parameter from time on task. A method has been laid out, and an illustration has been given. The illustration looked at just three items, all of which had been addressed by the same pupils. Extending the method to cover all of the possible items, which might come up in a simple math test, will either require a very large sample of student-item interactions, or the development of a system, which does not require exactly the same pupils to address every item.

Jonathan Hippisley
Email: jhipp -/at\- softway.org

References

Hippisley J (1999) Looking at Data from an Interactive Arithmetic Test from the Perspective of a Probabilistic Model. Education Research & Perspectives, (25)2, 59-67.

Khoshnevisan D (2007) Probability Graduate Studies in Mathematics, 80. American Mathematical Society.

L'Annunziata M (2012) Handbook of Radioactivity Analysis. Academic Press.

Rasch G (1960) Probabilistic models for some intelligence and attainment tests University of Chicago Press, Chicago


A Method of Estimating the Item Parameter from Time on Task. Jonathan Hippisley … Rasch Measurement Transactions, 2012, 26:3 p. 1380-2


Please help with Standard Dataset 4: Andrich Rating Scale Model



Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
May 26 - June 23, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 30 - July 29, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
July 31 - Aug. 3, 2017, Mon.-Thurs. Joint IMEKO TC1-TC7-TC13 Symposium 2017: Measurement Science challenges in Natural and Social Sciences, Rio de Janeiro, Brazil, imeko-tc7-rio.org.br
Aug. 7-9, 2017, Mon-Wed. In-person workshop and research coloquium: Effect size of family and school indexes in writing competence using TERCE data (C. Pardo, A. Atorressi, Winsteps), Bariloche Argentina. Carlos Pardo, Universidad Catòlica de Colombia
Aug. 7-9, 2017, Mon-Wed. PROMS 2017: Pacific Rim Objective Measurement Symposium, Sabah, Borneo, Malaysia, proms.promsociety.org/2017/
Aug. 10, 2017, Thurs. In-person Winsteps Training Workshop (M. Linacre, Winsteps), Sydney, Australia. www.winsteps.com/sydneyws.htm
Aug. 11 - Sept. 8, 2017, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Aug. 18-21, 2017, Fri.-Mon. IACAT 2017: International Association for Computerized Adaptive Testing, Niigata, Japan, iacat.org
Sept. 15-16, 2017, Fri.-Sat. IOMC 2017: International Outcome Measurement Conference, Chicago, jampress.org/iomc2017.htm
Oct. 13 - Nov. 10, 2017, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 5 - Feb. 2, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 10-16, 2018, Wed.-Tues. In-person workshop: Advanced Course in Rasch Measurement Theory and the application of RUMM2030, Perth, Australia (D. Andrich), Announcement
Jan. 17-19, 2018, Wed.-Fri. Rasch Conference: Seventh International Conference on Probabilistic Models for Measurement, Matilda Bay Club, Perth, Australia, Website
April 13-17, 2018, Fri.-Tues. AERA, New York, NY, www.aera.net
May 25 - June 22, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 29 - July 27, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 10 - Sept. 7, 2018, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 12 - Nov. 9, 2018, Fri.-Fri. On-line workshop: Practical Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
The HTML to add "Coming Rasch-related Events" to your webpage is:
<script type="text/javascript" src="http://www.rasch.org/events.txt"></script>

 

The URL of this page is www.rasch.org/rmt/rmt263d.htm,

Website: www.rasch.org/rmt/contents.htm,