The PAPERs Podcast

Selection for residency can be really hard and tricky. In this episode we discuss this paper that has done a scoping review to find out what perspectives there are for PGME selection.

Background

A key principle I have for education leadership is: spend time in choosing the people on your team.  Resident selection is one example (of many) of this.  Fortunately, at the point of selecting a resident for inclusion in a residency training program, it’s pretty hard to go wrong.  The system has already achieved maximum selection bias – choosing bright(brilliant), hardworking individuals, who understand the context and expectation of the profession of medicine.  Yet, selecting residents is hard.  In Canada it is not a manageable task.  The size of the applicant pool is too big – and we only have 17 medical schools.  I can’t even begin to imagine the challenge facing larger jurisdictions with candidate pools 10x larger.  And we haven’t begun to even discuss issues of unconscious / institutional bias, imbalance of supply/demand, south:north migration, travel (the costs and environmental impact of travelling to programs for in-person interviews), nor the questionable predictive validity (i.e. lack thereof) of a half day series of interview questions about “a time you failed,” or “what is your greatest weakness – and being a perfectionist is NOT an answer FYI” to determine a candidates fit into a program for the next 5 years of their life.  Is this process not simply doomed? 

I kinda feel like Princess Leia, signaling Obi Wan Kenobi in Episode 4.  “Help me, research all-stars, you’re my only hope.” Enter Caretta-Wyer et al.

Purpose

From the authors:

“Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature in order to explore practical  implications and present a research agenda.”

Methods

The authors conducted a scoping review- an appropriate synthesis they argue because of the explosion of the relevant literature with a lack of consensus and coherence.  If you’re looking for a clear, easy to follow, and classically reference methods section on scoping reviews, this manuscript has it. I won’t belabour the process.  We’ve been there, done that on The PAPERs Podcast.

PubMEd, EMBASE, Web of Science, ERIC and Google Scholar were all searched.  And all types of PGME papers (including editorials) were included. Thematic analysis (using Braun and Clarke’s approach) was used to identify themes.

From 4,200+ articles, 2,273 were eligible for inclusion.  Of these 100 articles were included in the review, 30 selected randomly, the 50 to enhance variability (by geography, publication date and context). When no new themes were identified an additional 20 manuscripts were reviewed, at random to confirm no new themes.

Results/Findings

Nearly 75% of manuscripts have been published in the last 10 years with 2/3 coming from the USA.  Of the reviewed papers, very few studies evaluated the effectiveness of the process nor identified underlying processes that influence selection (i.e., why does it work?)

The 5 perspectives on PGME selection (i.e., what is valued) are:

Using metrics to predict performance.

• What is the predictive validity of grades/scores on milestone reports/rank order lists etc.

• What are the measures of professionalism, communication (e.g., non-cognitive competencies) etc.?

• How to not select applicants requiring remediation

Competitive comparison to find the best applicant.

• Normative identification of top performers from past (e.g., medical school) contributions, NOT identification of FUTURE high performers.
• Heterogenous data makes this challenging, leading to calls for standardized assessments.

Finding the right “fit” between applicant and program

• Requires transparency between program AND applicant. (i.e., what is offered and what is desired)
• Attention to job analysis to specifically articulate the elements and characteristics to perform within the specialty and program.

Ensuring diversity and mitigating bias

• Focus on the end goal of a diverse work force to address societal needs.
• Addressing upstream bottlenecks that limit diverse applicant pools.

• Addressing conventional metrics that house bias (e.g., letters of reference, interviews etc.)

Optimizing the logistics of the selection process

• For programs: Revamp labour-intensive, non-scalable processes as the size of the applicant pool outstrips a program’s resources.
• For applicants: lack of standardization and transparency from programs enhances candidate anxiety.

Conclusions

From the authors:

“This scoping review provides insight into the value statements …  to describe postgraduate selection within the literature. These value statements provide a window into the assumptions and subsequent implications … It is clear that comparison of applicants is key to each value statement; however, each of these approaches centers a different stakeholder group at the expense of others.”

Comments

This is a RIME (research in medical education) paper.  A special, annual edition of Academic Medicine drawn from selected research presented at the AAMC meeting.  If you want to get a tight compendium of the hot topics in HPE, this annual, curated, peer-reviewed set of papers is a great place to start – either in print or live at the conference.