Standard setting using programmatic thinking for small cohort performance-based exams

Setting standards for determining competent performance on examinations is well-established in medical education. Establishing a conceptual boundary that differentiates competent from non-competent test-takers typically involves either expert judgement based on test item characteristics (e.g. Angoff), or an empirical process based on examinee performance (e.g. Borderline regression). Standard-setting procedures are a mandatory component of assessment processes in Australian specialist medical training programs. However, small specialist medical Colleges cannot implement existing standard-setting methods for performance-based examinations. Angoff-type methods are often not suited for performance examinations and empirical methods are unsuitable for small cohorts. A reform of assessment processes undertaken by the Royal Australian College of Dental Surgeons (RACDS) Oral and Maxillofacial Surgery (OMS) final examinations required implementation of appropriate standard-setting processes; however, the chosen method had to work with small candidate numbers and small numbers of examiners to support the process. [Author abstract]