Effect of trial items on candidate performances in a large-scale postgraduate medical selection test

The Graduate Australian Medical School Admissions Test (GAMSAT) is a cognitive test developed by the Australian Council for Educational Research (ACER) for the Consortium of Graduate-entry Medical Schools. GAMSAT consists of two writing tasks and two multiple-choice (MC) sections: Reasoning in Humanities and Social Sciences (75 items), and Reasoning in Biological and Physical Sciences (110 items). In each administration, each of the two MC sections includes different test booklets with the same cored items but different sets of trialled items. The trial item sets have been aimed to be equivalent in contents and difficulty levels across the test booklets. This study is designed to investigate effects of trial items in candidate test score (produced by cored items) and the stability of item difficulty of the cored items across the GASMAT test booklets (especially the ones locate right after the trial items). GAMSAT 2017 data is used.  IRT Three-faceted Rasch models, including candidate ability, test booklet difficulty effect and item difficulty have been implemented. Main results show that the aggregate data fits well to the Rasch model. There is a small variation in test booklet difficulty in both GAMSAT sections. Additionally, the variation of difficulties of the cored items by the test booklets does not yet show dependence on their positions (before or after the trial items). The results indicate a slight impact of trial items on candidate performances in GAMSAT.