Automatic Scoring of Items in PISA 2012 Problem Solving

The OECD PISA 2012 Problem Solving assessment was delivered on-line and exploited the capabilities of the computer by including many problems where students were required to interact with a virtual scenario. The actions taken by students – clicking buttons, dragging objects etc – were recorded in log files during testing. For many items these logs were used to automatically score the item. In some cases the strategy exhibited by the students as they tackled a problem contributed to the scoring, in addition to whether they managed to solve the problem. In this paper I give an overview of the PISA 2012 Problem Solving assessment; describe the types of interactive problems used in the assessment; present sample items to illustrate how students interact with the problem scenarios; and explain how actions relating to strategies were captured, interpreted and scored automatically.