Student Personality and Test Objectivity

How can we know when a single achievement test result is valid for a particular student? We can't. The universally-applicable uncertainty principle applies in several ways: we cannot determine the effect on the measured by the measuring instrumentation from a single test result. However, in some camps, there is a belief that it is possible to tell that a test produced an invalid measure of a student's performance by perusing the pattern of responses to test items. Is this an application of science or an attractive mythology? This is a serious question that has important implications for kids, teachers, and testing programs.

Is calculating "student fit" an example of detecting a difference? A discrepancy between what and what? Difference between students' item-by-item performance and the item-by-item relationships specified by Rasch mathematics? Student personalized test performance being different from behavior we expected to happen on a purely probabilistic basis? Difference between what a student did on a test and what that student should have done under more favorable sampling conditions?

It is always troubling when something done with numbers is equated to human behavior or misbehavior, if the connection between them is an explanation. Explanations never prove anything; they just make people say "Oh! I see." Since William Fisher has done such an artistic in-service on the contextual nature of seemingly objective observations of reality, such as response patterns, it might be appropriate to back off and consider the sources of the response patterns in tests and the way they are observed by psychometricians. For ground rules let's concern ourselves with the "what" and "how" questions of science and leave the "why" questions to philosophy.

Consider the mythology surrounding the traditional item discrimination index calculation and its use, without regard to the reality of curriculum pacing. What does this kind of "discrimination" really mean? Information about the wide or narrow range of the curriculum over which an item is supposed to reveal student knowledge is embodied in instructional programs. This range is only superficially, rather than causally, represented by high and low groups in some specified population.

Another myth is that some items should count more than others because they are mathematically related to high and low groups. These examples of overextended, tangential reasoning seem plausible enough to be swallowed by some statisticians.

In contrast, a different kind of background information comes from the practice, some years ago, of providing teachers with test information that incorporated information about other related characteristics of their students. The number of items students got correct and the number they attempted on a speeded verbal test were included in the teacher's report. To get a derived score, the number correct was divided by the number tried and this was then multiplied once more by the number correct. All scores were reported in standard notation based on the district population. Although it was not clear whether this "score" reflected study skills, a turtle-or-hare approach to problem solving, or was just equated for time constraints, it was clear that the teachers recognized and liked the way this information reflected their own assessment of the students approach to schooling. Some students worked carefully and got all of the few items they tried correct. Some raced ahead to increase the total number correct and take the penalty of more errors. Both behaviors reflected students' ways of working - their achievement mode. This derived "score" was more useful in reflecting ability to do school work on a day-to-day basis than any other we could derive from the data.

We know that test validity is specific to a student. We assume that validity differs across students taking a test. And, although we can't know their covert behaviors, we assume each student is unique in the combination of learning style, background experiences, cortical cycling speed, learned social responses, interest in different aspects of the curriculum, success or failure with schooling, and other personality characteristics, as well as in transitory attitudes toward school, testing, teachers, and others. Can't we, then, simply say that, whatever responses a student makes, they reflect ability to answer questions in this situation? Scores taken previously and in the future can be the data to use in deciding to delete a score as aberrant.

Consider also that validity is actually made up of two separate components; instrument validity is one and the student component, what we sometimes call reliability, is the other. Like heredity and environment, neither can exist without the other, but we can discuss their contributions individually. The personal part of test taking can only be guessed. Our perception of its manifestation is biased by many experiences and our learned response of searching for a statistic when all else fails. Why not leave reliability to whatever unknown factors might be exerting influence?

We know that learning for an individual should not be expected to progress in a linear manner or at a constant pace. We know that reinforcement of different learnings in the school situation varies and its effects on different individuals vary. And we know that teaching some lessons make earlier learnings evaporate. For example, students who formerly knew how to correctly add a number to zero will often incorrectly answer later that a number and zero equal zero after they have been taught that a number multiplied by zero equals zero.

Low raw scores are a different matter. They raise a different question - poor test construction. First, get rid of ambiguous items. Second, use a different test. When students get less than a third of the items correct on a test, it is due to faulty test assignment procedures and can be corrected by retesting with a lower level test. Inevitable idiosyncrasies in teaching and learning must never be penalized in a misguided effort to report higher test reliabilities.

George Ingebo
Portland Public Schools

Student personality and test objectivity. Ingebo G. … Rasch Measurement Transactions, 1990, 3:4 p.86




Rasch Publications
Rasch Measurement Transactions (free, online) Rasch Measurement research papers (free, online) Probabilistic Models for Some Intelligence and Attainment Tests, Georg Rasch Applying the Rasch Model 3rd. Ed., Bond & Fox Best Test Design, Wright & Stone
Rating Scale Analysis, Wright & Masters Introduction to Rasch Measurement, E. Smith & R. Smith Introduction to Many-Facet Rasch Measurement, Thomas Eckes Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences, George Engelhard, Jr. Statistical Analyses for Language Testers, Rita Green
Rasch Models: Foundations, Recent Developments, and Applications, Fischer & Molenaar Journal of Applied Measurement Rasch models for measurement, David Andrich Constructing Measures, Mark Wilson Rasch Analysis in the Human Sciences, Boone, Stave, Yale
in Spanish: Análisis de Rasch para todos, Agustín Tristán Mediciones, Posicionamientos y Diagnósticos Competitivos, Juan Ramón Oreja Rodríguez

To be emailed about new material on www.rasch.org
please enter your email address here:

I want to Subscribe: & click below
I want to Unsubscribe: & click below

Please set your SPAM filter to accept emails from Rasch.org

www.rasch.org welcomes your comments:

Your email address (if you want us to reply):

 

ForumRasch Measurement Forum to discuss any Rasch-related topic

Go to Top of Page
Go to index of all Rasch Measurement Transactions
AERA members: Join the Rasch Measurement SIG and receive the printed version of RMT
Some back issues of RMT are available as bound volumes
Subscribe to Journal of Applied Measurement

Go to Institute for Objective Measurement Home Page. The Rasch Measurement SIG (AERA) thanks the Institute for Objective Measurement for inviting the publication of Rasch Measurement Transactions on the Institute's website, www.rasch.org.

Coming Rasch-related Events
May 17 - June 21, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 12 - 14, 2024, Wed.-Fri. 1st Scandinavian Applied Measurement Conference, Kristianstad University, Kristianstad, Sweden http://www.hkr.se/samc2024
June 21 - July 19, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Winsteps), www.statistics.com
Aug. 5 - Aug. 6, 2024, Fri.-Fri. 2024 Inaugural Conference of the Society for the Study of Measurement (Berkeley, CA), Call for Proposals
Aug. 9 - Sept. 6, 2024, Fri.-Fri. On-line workshop: Many-Facet Rasch Measurement (E. Smith, Facets), www.statistics.com
Oct. 4 - Nov. 8, 2024, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
Jan. 17 - Feb. 21, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
May 16 - June 20, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com
June 20 - July 18, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Further Topics (E. Smith, Facets), www.statistics.com
Oct. 3 - Nov. 7, 2025, Fri.-Fri. On-line workshop: Rasch Measurement - Core Topics (E. Smith, Winsteps), www.statistics.com

 

The URL of this page is www.rasch.org/rmt/rmt34f.htm

Website: www.rasch.org/rmt/contents.htm