Evaluation of booklet effect in PISA mathematics across countries

In PISA 2003, 13 main linked test booklets were constructed by item cluster rotation design. Each student was randomly assigned one of the test booklets. Each of the selected clusters was located in four of the test booklets in different positions. There were seven mathematics clusters with a total of 85 items. Booklet effect was identified and was used to adjust the measure of student performance in PISA (see PISA 2003 technical report; OECD, 2004). This study was designed to explore which factors could account for this booklet effect in mathematics across 41 PISA countries. A three-faceted partial credit model was implemented by ACER ConQuest to obtain the estimates of booklet effect indices for each country. Then these values were examined together with three factors: number of the mathematics items, proportion of the different item format (open-ended versus multiple-choice), and the average of the item position in the test booklet. Results showed that item position was the most significant cause of the test booklet effect. Additionally, the different number of mathematics items in a test booklet could influence the effect index. However, the proportion of different item formats did not show a significant impact on test booklet effect in this data. These findings suggest that when assembling linked tests, besides balancing the proportion of different item formats, contents and difficulty, items positions should be carefully considered.