From log file analysis to item response theory: an assessment template for measuring collaborative problem solving

Recent economic, educational and psychological research has highlighted shifting workplace requirements and the change that is required in education and training to equip the emerging workforce with the skills for the 21st century. The emergence of these demands highlights the importance of new methods of assessment. An earlier study, ATC21S, pioneered assessment of individuals’ collaborative problem solving (CPS). The study represented a major advance in educational measurement, although the issue of efficiency, reliability and validity remained to be resolved. This study addresses some of these issues by proposing and developing an assessment template for measuring CPS in online environments. The template presented, from conceptualisation to implementation, centres on its generalizable application. The first part of the template outlines task design principles for the development of CPS tasks. The second part of the template presents a systematic process of identifying, coding and scoring behaviour patterns in log file data generated from the assessment tasks. Item response theory is used to investigate the psychometric properties of these behaviour patterns. Behavioural indicators are presented that are generalizable across students, CPS tasks and assessment sets. The goal of this study is to present an approach that can inform new measurement practices in relation to previously unattended latent traits and their processes. The assessment template provides an efficient approach to development of assessments that measure the social and cognitive subskills of collaborative problem solving.