#67 – First we build the AI, then the AI builds us
Episode host: Lara Varpio.
In this episode, Lara leads a conversation about AI and the current body of knowledge about AI that is growing rapidly in Medical Education. Everything you need to know about AI in MedEd is in this paper!
Episode 67 transcript. Enjoy PapersPodcast as a versatile learning resource the way you prefer—read, translate, and explore!
Episode article
Gordon, M., Daniel, M., Ajiboye, A., Uraiby, H., Xu, N. Y., Bartlett, R., Hanson, J., Haas, M., Spadafore, M., Grafton-Clarke, C., Gasiea, R. Y., Michie, C., Corral, J., Kwan, B., Dolmans, D., & Thammasitboon, S. (2024). A scoping review of artificial intelligence in medical education: BEME Guide No. 84. Medical Teacher, 46(4), 446–470.
Episode notes
Here is the notes by Lara Varpio.
Background to why I picked this paper
The paper I picked today is one I found when I was working with a collaborator who is interested in AI and how it is being used in Medical Education. It is the perfect overview paper for all things AI in Med Ed! The paper is entitled A Scoping Review of Artificial Intelligence in Medical Education. It is a BEME review published in 2024
Purpose of this paper
“This review aims to provide insights into the currentstate of AI applications and challenges within the full continuum of medical education.”
(Gordon et al., 2024)
Methods used in this paper
The authors conducted a scoping review… with a twist. They conducted a rapid scoping review. From what I could find in the literature about review methodologies, a rapid scoping review is quite similar to a scoping review, but there are a few key differences. One is that the project timeline is short. So if a scoping review is expected to take a year or so to do, then a rapid scoping review aims to be done in 4 months. Scoping reviews have several broad research questions; a rapid scoping review has fewer questions that are clearly specified, narrower. Scooping review has exhaustive and broad searchers; rapid scooping reviews have limits on the scope. Scoping reviews have data extraction approaches that focus on depth and the generation of knowledge; rapid scoping reviews are more tailored in their data extraction that is tailored to meeting specific aims.
This study was conducted within 16 weeks of its inception—so it is a rapid scoping review. They followed Arksey and O’Malley’s methodology. They chose not to use Levac’s recommendation for external stakeholder feedback because they had sufficient expertise on the author team to fulfill that role. They did have 16 authors so maybe…
They searched PubMed/MedLine, Emboss and Med Ed Publish. Pub med and Embase are two good databases that are often used in Med Ed literature reviews. The authors may not have included as many databases as we’d expect, but that’s fine — it is a rapid review. That’s a limit we’d expect to see imposed. They did a hand search of identified articles and added manuscripts that were referenced that were deemed relevant.
They included pretty much every paper published that addressed AI in med ed—excluding only articles that dealt with AI used for diagnostic purposes, clinical or organizational purposes, or that talked about AI only for research purposes They divided the corpus into 2 for data extraction. One group consisted of all the paper that were research reports or innovations. The second group consisted of perspectives and opinion papers. For the research and innovation papers, they developed and then used an extraction tool that collected information about study demographics and characteristics, about how the AI was used, about the kind of AI used, the rationale for the use of the AI, evaluation results or Kirkpatrick outcomes if relevant, and implications for the future. For the perspectives papers, they use the same data extraction tool but also added details about the rational for using AI, its applications framework and recommended topics for curriculum and research, and ethical issues.
Results/Findings
Once the authors did their search and added additional papers found via hand search, they had a corpus to review that consisted of 278 papers. They say a full appendix table of extracted data for all the papers had been uploaded to a repository—unfortunately, it seems that link is broken ☹
You won’t be surprised to hear that, while the first paper about AI was published in 1998, the real surge of papers has come out since 2018. If there were 11 papers published in 2018, that number was up to 57 in 2022 and up to 114 as of August 2023. About 70% of the papers were research or innovation reports, with the other 30% being perspectives and opinion papers. The focus of about 50% of the corpus was UME, 22% was GME, and 3 percent were CPD. With the remainder being some mix of levels. The papers were from 24 different clinical specialities and different basic science departments.
So, there are a TON of different segments to the results. The authors offered many ways of mapping the literature to see what is in there. So instead of me reading all that off to you, we’ll each take a section of the results that we thought was particularly interesting.
Ethics: My interest was piqued by the 14 papers in the corpus that addressed ethics. They expressed caution about the limitations of AI applications in medical education and about the need for educators to really grapple with the ethics of AI when we have yet to really figure out how AI is being used.
The paper lists offered some topics that should be covered in AI ethics education. Some I think are particularly important are algorithmic bias and equity. AI are trained by existing data, and we know those data are non-representative of all populations. This can exacerbate healthcare disparities if we just assume the AI is offering objective facts. It isn’t. It is offering facts generated off biased data sets, so those offerings are biased.
Another ethics concern must be transparency and informed consent. Physicians have a duty to inform their patients about how AI is being used in their treatment and care planning, and about how their data is being collected and used. And their anonymity is not iron clad. With more and more AI resources being developed, it is increasingly possible for people to be identified across platforms.
Conclusions
The paper ends by offering the FACETS framework which the authors suggest should be used to make sure that future manuscripts reporting on AI are comprehensive. This more comprehensive approach, they suggest, can support dissemination, replication, and innovation. The FACETS framework calls for these manuscripts to describe: the form of AI used, the AI use case (ie the end product, innovation, or outcome achieved by the AI), the context, the educational focus, the technology used, and the SAMR (ie the level of technological integration)
“The landscape of AI in medical education, as charted in this review, spans a wide array of stages, specialties, purposes and use cases, primarily reflecting early adaptation phases–only a few describe more in-depth employment for longitudinal or deep change. The proposed FACETS framework is a key outcome, offering a structured approach for future research and practice.”
0 comments