Factors that influence the difficulty of problem-solving items

This chapter presents a study undertaken by the authors using data from the PISA 2012 computer-based assessment of problem solving. It considered ten characteristics understood to influence the difficulty of items used in problem-solving assessment. Each item was rated to reflect the amount of each characteristic it possessed. The item responses from about 85 000 participants from 44 countries were analysed to obtain item response theory (IRT) estimates of item difficulties. The predictor characteristics were analysed in a number of ways, including a hierarchical cluster analysis, regression analysis with item difficulty as outcome variable and a principal component factor analysis. The main characteristics predicting difficulty seem to be: the complexity and type of reasoning skills involved in solving the problem; the amount of opportunity the solver is given to experiment or uncover hidden facets in a problem scenario (more opportunity to explore and experiment will make a problem easier); and the number and nature of constraints that a solution must satisfy (complex or conflicting constraints will make a problem more difficult).