The Collaborative Problem Solving (CPS) assessment, developed for the Programme for International Student Assessment (PISA) 2015, is an innovative computer-mediated test designed to measure students’ abilities to work effectively in team settings to solve problems. CPS is defined as the capacity of an individual to engage actively in a process where two or more agents share understanding, effort, and pool knowledge and skills to reach a solution (OECD, 2017). Unlike traditional assessments focusing on individual knowledge, CPS includes the social component of problem solving, capturing how individuals collaboratively communicate, negotiate, and organize during problem-solving tasks.

Framework and Dimensions

The CPS framework used in PISA 2015 crosses two dimensions—collaboration and problem solving—to create 12 specific skills organized in a 3-by-4 matrix. The three collaboration processes are:

Establishing and maintaining shared understanding among team members,

Taking appropriate action to solve the problem,

Establishing and maintaining team organization.

The four problem-solving processes adapted from PISA’s 2012 individual problem-solving assessment are:

Exploring and understanding the problem,

Representing and formulating the problem,

Planning and executing strategies,

Monitoring and reflecting on problem-solving activities (Graesser et al., 2017; OECD, 2017).

This yields 12 distinct skills such as initiating communication, coordinating roles, planning solutions collaboratively, and monitoring collective progress.

Method of Assessment

The assessment occurs via computer-based tasks where students interact with computer agents simulating other team members through chat and action-oriented interfaces (Graesser et al., 2017). These agents are programmed to behave like human collaborators and can generate goals, communicate messages, sense the digital environment, and adapt to changing problems and team dynamics in real-time. Test-takers engage in realistic scenarios requiring negotiation, cooperation, and shared decision-making to solve complex problems.

Psychometric and Practical Advantages

Using computer agents allows standardized, scalable assessment of collaboration skills without logistical challenges of assembling human teams for testing. The test focuses on the individual’s contribution to the team process, providing detailed data on communication quality, strategic planning, and teamwork behaviors. This approach integrates social and cognitive aspects, offering richer, more authentic data than traditional tests focused on individual knowledge recall (Graesser et al., 2017; OECD, 2017).

Challenges and Considerations

Although promising, CPS assessment has validity constraints because interacting with simulated agents differs from real human interaction. There are also challenges in ensuring equitable digital access and interpreting scores within diverse cultural contexts. Studies suggest that test results should be interpreted as reflecting collaborative problem-solving ability specifically within the context of interaction with computer agents under controlled assessment conditions (Herborn et al., 2020; Validation Study, 2022).

References

Graesser, A., D’Mello, S., & Hu, X. (2017). Complex problem solving in assessments of collaborative problem solving. Computers in Human Behavior, 76, 414-426. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6526432/

Herborn, K. (2020). The assessment of collaborative problem solving in PISA 2015. Computers & Education, 129, 101-111. https://doi.org/10.1016/j.compedu.2018.12.001

Organisation for Economic Co-operation and Development. (2017). PISA 2015 Assessment and Analytical Framework: Science, Reading, Mathematics and Financial Literacy (Vol. 1). OECD Publishing. https://www.oecd.org/pisa/pisa-2015-assessment-and-analytical-framework-9789264281820-en.htm

Validation of the PISA 2015 collaborative problem-solving competence assessment (2022). University of Manchester. https://research.manchester.ac.uk/en/studentTheses/validation-of-the-pisa-2015-collaborative-problem-solving-compete