A new role for item aficionados in high-stakes testing programs

The purpose of this paper is to point teachers, test analysts, and users of test results to the significance of student responses at the item level and considering what it is that each item purports to measure and actually measures before taking the evidence of a low score on a test − just a score derived from a collection of items − and coming to the seemingly obvious but not necessarily accurate conclusion that the student has no knowledge or understanding of the domain being tested. An item aficionado does not approach test items and test results at the level of abstraction of ability. Rather, the role is to identify what students can do and what they cannot do or have trouble with, to understand sources of item difficulty, and to critique tests.