#15 Digitization & Despair: Two Cases of Complex Change

June 6, 2023

Host: Jason Frank

Enjoy listening to us at your preferred podcast player.

Episode article

Humphrey-Murto, S., Makus, D., Moore, S., Watanabe Duffy, K., Maniate, J., Scowcroft, K., Buba, M., & Rangel, J. C. (2023). Training physicians and residents for the use of Electronic Health Records—A comparative case study between two hospitals. Medical Education, 57(4), 337–348. https://doi.org/10.1111/medu.14944

Openspotify

In this episode, Jason introduces a paper that uses case study methodology to look at complex educational interventions, and why so often they fail

Background

What is it about big, complex, educational interventions that make them so often fail? Sometimes, I am obsessed with and perplexed by this problem. New curricula, new teaching, new content, new settings, new assessments…All these are often set up to fail.

Another example of this is new technology, like a new e-portfolio or a new electronic health record (EHR). These two are notorious and dreaded phenomena.

I have PTSD from my experience. I am still pondering the lessons learned from implementing the EHR education in our hospital, and what I would do differently for a future complex educational intervention.

Enter, Dr. Humphrey-Murto et al. (Caveat: I work in the same institution as several of the authors and the first author is a friend…But she doesn’t know I am reviewing this paper…Yet.)

Purpose

The authors pitched that there is a gap in our understanding of how physicians are trained for EHR implementation, and this is holding back the world achieving EHR benefits. The authors set out to “describe and critically assess the interplay between educational activities and organizational factors that influenced EHR training and implementation across two different hospitals”.

Methods

Have you ever used case study methodology? I had to take dive into the shallow end of my methods sources to look for descriptions of use and rigour (see text box).

The authors performed a comparative qualitative case study using Singh & Sittig’s Socio-technical model of studying health information technology in complex adaptive healthcare systems. The latter has 8 elements:

Hardware & software
Clinical content
Human-computer interface
People
Workflows
Organizational
External rules
System monitoring

Case study methodology

Allows in-depth explorationsof complex phenomena in real life settings
Aka “naturalistic designs
Key authors Stake (1995) and Yin (2009)
Intrinsic Case studies: a unique phenomenon
Instrumental: a typical phonomenon
Collective: multiple instances studied togehter
Case studies can be used to explain, describe or explore a phenomenon in context
Can help develop or refine theories
Can be done through variety of epistemological lenses

They compared the implementation of the same EHR software at 2 teaching hospitals. The adult hospital chose “big bang” all-at-once implementation in 1 day, and the pediatric hospital chose “bit-by-bit” over 8 years(!).

They performed 1:1 interviews remotely with a purposive sample of physicians, residents, students and administrators. They used standard methods of transcription, coding for themes, and group interpretation. They were not allowed to review proprietary training materials for the study.

The study had research ethics board (REB) approval and they provided a table as their reflexivity statement.

Results/Findings

43 participants were interviewed until they “achieved sufficient information power”. There were no consistent differences between the 2 sites, so the data was analyzed in aggregate.

Some notable highlights:

Training provided was too general
There was limited transferability to daily work
Most learning was on the job and from peers, despite significant investment in corporate support (1200 trainers at the adult site!)
Majority of physicians felt unprepared to work
Majority of end-users felt strong negative emotional reactions (e.g. made to feel more incompetent)
Clinical workflows were often completely disrupted, and users felt blind-sided by this
Users employed workarounds
Focus was on computers > patients
Training was inadequate before, during, and after implementation
Corporate workforce transformation was needed.

Conclusions

The authors concluded that implementation of major complex projects like an EHR is not a software project, but a socio-technical organizational transformation. They provided recommendations on how to alleviate the significant impacts.

Comments

This paper highlights both case study methods and has lessons for big complex educational interventions

References

Crowe, S., Cresswell, K., Robertson, A., Huby, G., Avery, A., & Sheikh, A. (2011). The case study approach. BMC Medical Research Methodology, 11(1), 100. https://doi.org/10.1186/1471-2288-11-100

Malterud, K., Siersma, V. D., & Guassora, A. D. (2016). Sample Size in Qualitative Interview Studies: Guided by Information Power. Qualitative Health Research, 26(13), 1753–1760. https://doi.org/10.1177/1049732315617444

Singh, H., & Sittig, D. F. (2020). A Sociotechnical Framework for Safety-Related Electronic Health Record Research Reporting: The SAFER Reporting Framework. Annals of Internal Medicine, 172(11_Supplement), S92–S100. https://doi.org/10.7326/M19-0879

Stake, R. E. (1995). The art of case study research. Sage Publications.