The PAPERs Podcast

Background 

What was your assessment experience during your training? Was it useful? 

Recent decades witnessed a mountain of publications and efforts in all health professions directed at improving assessments. As documented by van der Vleuten and Schuwirth in their recent magnum opus about assessment, the trend is now to deploy assessments as a system that answers all the criticisms of recent years. Now we are seeing programmatic assessment, with more direct observations, more feedback, less emphasis on high-stakes written tests. So now we have solved all the assessment problems, right? 

Wrong! Now we have rising complaints that all these direct observation WBAs are impossible to implement, are a “burden”, harm patient care efforts, or worst of all—cause mental health issues in learners. Is this era of programmatic assessment doomed? Is this just a fidelity problem with how we do implementation, or is there a fundamental problem with our designs? 

Purpose 

Enter Phinney et al (the team includes meded gurus Karen Hauer and Bridget O’Brien, a resident, and 2 med students!), from UCSF. They implemented a new, completely formative, WBA direct observation tool to clinical clerkships during the COVID pandemic. In this study, they sought to “understand the role of a WBA tool in facilitating feedback” using cultural historical activity theory (CHAT).  

About CHAT 

Aside: I am intrigued by CHAT, which I only paid attention to last year. Originating with Russian psychologists Vygotsky & Leontiev, it has recently been popularized by Yrjo Engestrom from the University of Helsinki and others. CHAT is “a theoretical framework that helps to understand and analyse the relationship between the human mind and activity.” (Wikipedia) The framework is friendly to social psychology and constructivist epistemologies. CHAT describes activity systems: interactions between tools, users, rules, communities, division of labour, the purpose of an activity, and the results of the activity. The framework elicits tensions between the various elements.  

Methods

For this study, the authors conducted semistructured interviews of medical students who used the new WBA over 2 cohorts that used different versions of the tool. The WBA tool included drop-down lists, an “entrustment” supervision scale, and a text box. In Year 1, the tool had to be filled out by a preceptor on a desktop computer. In year 2, students could fill out their own WBAs, and anyone could scan a QR code on the student’s badge to start an assessment. Students were told that the forms were not to be used for their marks or promotions. Only the learner, the preceptor, and the student’s longitudinal coach could see a collection of these forms.  

CHAT was used to construct the interview guide and as a framework for coding transcripts. Subsequently, transcripts were reviewed to identify tensions that arose with the implementation of the new WBA approach.  

The authors addressed reflexivity by journaling, keeping notes, and discussing. UCSF exempted this study from ethics.  

Results/Findings 

In year 1, there were 6818 WBAs by supervisors only. 18/168 students participated in the study. In year 2, there were 9039 WBAs, but only 18% from supervisors. The rest were self-ratings by students. 25/167 students participated.  

Identified Tensions included: 

1. Misinterpretation / anxiety about this WBA being summative.  

2. Cumbersome tool design in year 1 was a barrier to quality feedback. 

3. Avoidance of burdening busy clinical supervisors. 

4. Perception of WBAs as useless checking boxes.  

5. Learning environment-specific WBA cultures.  

Tensions evolved over time.  

Conclusions 

The authors concluded that CHAT gave unique systemic insights into what works and what doesn’t when implementing an innovation in HPE. Managed well, these tensions can drive system transformation and growth (Engestrom’s theory of expansive learning).  

They recommended: 

1. Design separate platforms for purely feedback assessments. 

2. Positive assessment culture must be fostered for success. 

3. Promote learner autonomy. 

4. Use technology to facilitate real-time feedback.  

5. Develop trusting, long-lived relationships between preceptors & students.  

Comments

This paper illustrates the use of the CHAT framework to provide powerful and actionable insights into the dynamics of an HPE activity in context.  

This is a great example of using a theoretical framework to explain the results. 

This paper provides key insights for those seeking to implement WBA or any other major innovation.  

References

Engestrom, Y. (2000). Activity theory as a framework for analyzing and redesigning work. Ergonomics, 43(7), 960–974. https://doi.org/10.1080/001401300409143

Engeström, Y., & Pyörälä, E. (2021). Using activity theory to transform medical work and learning. Medical Teacher, 43(1), 7–13. https://doi.org/10.1080/0142159X.2020.1795105

Holmboe, E. S., Sherbino, J., Long, D. M., Swing, S. R., & Frank, J. R. (2010). The role of assessment in competency-based medical education. Medical Teacher, 32(8), 676–682. https://doi.org/10.3109/0142159X.2010.500704

Kogan, J. R., & Hauer, K. E. (2020). Sparking Change: How a Shift to Step 1 Pass/Fail Scoring Could Promote the Educational and Catalytic Effects of Assessment in Medical Education. Academic Medicine, 95(9), 1315. https://doi.org/10.1097/ACM.0000000000003515

Larsen, D. P., Nimmon, L., & Varpio, L. (2019). Cultural Historical Activity Theory: The Role of Tools and Tensions in Medical Education. Academic Medicine, 94(8), 1255. https://doi.org/10.1097/ACM.0000000000002736

Schuwirth, L. W. T., & van der Vleuten, C. P. M. (2020). A history of assessment in medical education. Advances in Health Sciences Education, 25(5), 1045–1056. https://doi.org/10.1007/s10459-020-10003-0