The PAPERs Podcast

Episode host: Linda Snell.

Episode 73 transcript. Enjoy PapersPodcast as a versatile learning resource the way you prefer- read, translate, and explore!

---

Episode article  

Wood, T. J., Daniels, V. J., Pugh, D., Touchie, C., Halman, S., & Humphrey-Murto, S. (2024). Implicit versus explicit first impressions in performance-based assessment: Will raters overcome their first impressions when learner performance changes? Advances in Health Sciences Education, 29(4), 1155–1168.

Episode notes

We all use first impressions—those immediate, instinctive judgments formed within seconds of meeting someone. These judgments, often referred to as “thin slices” or “zero acquaintance judgments,” are based on incomplete information and processed automatically with little conscious awareness. They’re a natural part of human interaction, but in the high-stakes context of health professions education, their impact can be profound.

Earlier research in the subject

The authors did an extensive description of earlier research in the subject:

First impressions matter in health professions education

Predictive of subsequent judgments: First impressions often influence later evaluations, such as teacher ratings, OSCEs, or Multiple Mini Interviews (MMIs). 

Performance changes and bias: A learner’s initial performance may lead to over- or under-rating if that performance later improves or declines. 

Context matters: Most prior research on first impressions has been conducted in exam contexts, such as OSCEs, rather than in real-world clinical settings. 

Implications for CBME: In competency-based medical education (CBME), first impressions play a role in entrustment decisions. Early judgments (thin slices) and presumptive entrustment (non-acquaintance judgments) can shape critical decisions about learners. 

Explicit vs. implicit judgments: The act of making a first impression explicit has been shown to influence subsequent ratings compared to situations where evaluations remain implicit. 

Purpose 

The paper aimed to explore two key questions: 

1. To determine what degree raters are influenced by their first impressions when learner performance changes in a naturalistic context (rather than in an OSCE).  

2. To determine if making a first impression judgment explicit affects subsequent ratings compared to a more natural (implicit) rating process. 

Methods 

The authors designed an experimental study with clinical supervisors from internal medicine, emergency medicine, and family medicine at two Canadian universities. Participants were divided into two groups:

The study compared two groups of clinical supervisors

Explicit First Impressions: Supervisors in this group paused the video after 60 seconds to provide a “First Impression Global Rating” (FIGR). This step required them to consciously evaluate their initial impression of the learner’s performance.

Implicit First Impressions: Supervisors in this group viewed the full video without interruption, allowing their first impressions to remain unspoken and implicit.

Analysis of Data: 

The study employed multiple statistical analyses, focusing on correlations between different ratings:

• The internal consistency of Mini-CEX item ratings across the six cases.
• Comparisons of FIGR (for the explicit group) with final global ratings.
• Comparisons of item ratings and global ratings across the explicit and implicit groups. 
• Performance changes (e.g., weak-to-strong vs. strong-to-weak) and their influence on final ratings. 

The sample size was set at 50 participants, divided equally between the explicit and implicit groups. However, the final analysis included data from 45 supervisors (25 in the explicit group and 20 in the implicit group), slightly below the target. While this minor shortfall introduces a theoretical risk of a Type II error (failing to detect a difference where one exists), the authors note that the observed results are robust and unlikely to be significantly altered by additional participants. 

Reflections from the Hosts: 

• The study’s design was rigorous, but the slightly smaller sample size and reliance on a simulated setting may limit generalizability to authentic clinical environments.
• The analysis reinforced prior findings, showing that supervisors adjust their judgments when presented with new performance data.
• The reliance on the Mini-CEX, while standard in clinical education, highlights known limitations of the tool, such as its tendency to produce highly correlated ratings across domains (suggesting potential Halo effects).

Results/Findings 

The study yielded several noteworthy findings, contributing to the understanding of how first impressions influence workplace-based assessments:

Judgment Adjustments:

• In cases where a learner’s performance evolved (weak-to-strong or strong-to-weak), raters adjusted their final global ratings to align with the observed performance changes.
• This suggests that first impressions, while impactful, are not fixed and can be recalibrated based on new information.

Explicit vs. Implicit First Impressions:

• No significant differences were found between the final global ratings of the explicit and implicit groups.
• This indicates that requiring raters to articulate their first impressions explicitly did not influence their subsequent evaluations any more than natural, implicit processes.

Consistent Performance:

For learners with consistent performance (e.g., strong throughout), both explicit and implicit raters provided similar ratings, reinforcing the stability of evaluations when performance does not fluctuate.

Bias Toward Initial Performance:

• There was a subtle trend suggesting that initial impressions may have a greater impact when the starting performance is weak, as raters tended to adjust less drastically upward.
• This finding aligns with the psychological principle that negative first impressions can be harder to overcome.

PAPER Clips

Reassurance for Educators: The results are reassuring, showing that thoughtful clinical supervisors can adapt their judgments based on observed changes in learner performance, countering concerns about the rigidity of first impressions.

Operational Implications: The study supports the idea that explicit first impression ratings are unnecessary in routine assessment processes, as implicit judgments naturally evolve with new evidence.

Questions for Future Research:

• Demographic Data: The study did not collect demographic data about the raters (e.g., years of experience, clinical specialty), missing an opportunity to explore whether these factors influence first impressions or the ability to adjust them. Collecting such data is relatively straightforward and could add valuable depth to future studies.
• Mixed Methods Approach: Including qualitative elements, such as interviews or think-aloud protocols, could help answer some of the “why” questions that the study could not explore. For instance, why do some raters adjust their judgments more than others? What cognitive processes are involved in overcoming a poor initial impression?
• Social and Implicit Bias: Can first impressions influenced by social or cultural biases be mitigated through interventions, and how might these biases affect the calibration of judgments?Can implicit biases related to social or cultural factors be mitigated through targeted interventions?
• Authenticity of Settings: Extending the research to real-world clinical environments could reveal additional complexities not captured in the controlled simulation.

Reflections on Practical Applications:

Entrustment Decisions: First impressions, especially in competency-based medical education, can shape initial entrustment decisions. The ability to recalibrate judgments provides a safeguard against premature biases.

Learner Handover: The findings suggest that non-acquaintance judgments (e.g., a handover summary) may not strongly influence subsequent ratings, offering some reassurance about the objectivity of assessments over time.

Limitations of the Mini-CEX: While widely used, the study highlights the need for more nuanced tools to capture complex workplace dynamics and reduce potential Halo effects in ratings.45 raters 

---

Transcript of Episode 73

This transcript is made by an autogenerated text tool and some manual editing by the Papers Podcast team. Read more under “Acknowledgment”.

Jason Frank, Lara Varpio, Linda Snell, Jonathan Sherbino.

Start

[music]

Lara Varpio: Hi, everybody, and welcome to another episode of the Papers Podcast. It is our joy to come into your phone and your earphones every week and share with you a little summary, a discussion of some of the medical education literature that we think is actually worth a little bit of time and reflection.

Lara Varpio: And so with that, of course, Of course, it wouldn’t be a conversation without some friends. So first, let me say, hey, Jon, how you doing?

Jonathan Sherbino: Good, Laura. I like your ballsy Canadian tuxedo wear. Yeah. Do you want to explain for the audio segment what you have on?

Lara Varpio: So for those of you who aren’t in the know, a Canadian tuxedo is a flannel shirt with checks. And I have one of my favorites on today because I forgot we were doing recordings. And no makeup, no hair, and Canadian tuxedo. I’m just saying.

Jonathan Sherbino: It’s my favorite version of you.

Lara Varpio: Oh, bless you.

Lara Varpio: And Jason, where’s your tuxedo, dude?

Jason Frank: These are my jammies. So this is where I’m always in. Yeah. Hi, everybody. It’s great to be here.

Lara Varpio: And today, my friend, Linda, it’s always good to see you. You have picked a selection for us about first impressions. So I’m really excited because my Canadian tuxedo is definitely making a first impression.

Linda Snell: Definitely. It is plaid shirts and all. So I’ll start by saying this article is. Called Implicit Versus Explicit First Impressions in Performance-Based Assessment, colon, Will Raiders Overcome Their First Impressions When Learner Performance Changes? It’s in 2024.

Linda Snell: It advances in health sciences education with a cast of stars, including Tim Woods, Vijay Daniels, Deborah Pugh, Claire Touchie, Samantha Halman, Susan Humphrey-Murto. All colleagues, most of them are colleagues of Jason. So my title for this is, Does My First Impression Matter? And Does It Matter If I Think About It?

Linda Snell: So let me ask you very briefly to describe a first impression where maybe it wasn’t confirmed as you got to know an individual. I’ll go Lara, Jason, Jon.

Lara Varpio: Linda, I don’t have one about a person, individual that I knew, but my favorite example of first impressions gone awry. Did you all used to watch Britain’s Got Talent? One of the earliest seasons, there was a woman who was a singer. And when they first showed her, she just looked like a normal person, just like any of us.

Lara Varpio: And then she sat up on stage and you could tell that the Simon Cowell, one of the judges was a little bit unimpressed with how this was already going. And then she opened her mouth and like a… Voice of an angel just like, bam, just stopped everybody dead. I love it when that happens. So that’s my favorite example of first impressions gone awry.

 Jonathan Sherbino: Her name was Susan something, I think.

Lara Varpio: Yeah, it was.

Jonathan Sherbino: Susan something.

Linda Snell: Something. She got a recording out of it too, I think.

Lara Varpio: And while you discuss it, I will find her name.

Jason Frank: Didn’t she perform in sweatpants the first time or something like that? Like totally casual.

Linda Snell: Going back to first impressions, Jason.

Jason Frank: This happens every week with learners. You can meet someone and hear the beginning of some case or watch some piece of something, and then over the course of working with them, you realize that they’re actually in some other place or they’re just growing in front of you.

Jason Frank: Just recently, I worked with somebody, and it was actually their first day in Ottawa with us in the emergency department, and they were struggling with the electronic medical record and they were struggling with how things work here, and that made them look okay.

Jason Frank: But not superstar. And then I worked with them again a month later and they were superstars. So this happens all the time. All of us who are raters have to be really careful about documenting what we see and not comparing them to some other archetype.

Lara Varpio: Her name was Susan Boyle. Her name was Susan Boyle and Boyle made over 5 million British pounds. And by today’s money, that’s 8.5 million with the release of I Dreamed a Dream and her success continued her second album. Was one of the first to top both the UK and US album charts with two different albums in the same year.

Linda Snell: Boom.

Jason Frank: Lava, you can sing too. Go ahead.

Lara Varpio: No, that’s not me. No, no.

Jonathan Sherbino: Let’s just keep going. So my first impression was of the three of you, I thought, wow, such beautiful, smart, intelligent people.

Lara Varpio: Oh, no. Careful, he wants money or he wants a favor.

Jonathan Sherbino: Look, the reader decided that first impression held true 20 years later.

Jason Frank: That’s the before.

Jonathan Sherbino: Here’s my first impression story, Linda, is I’ve done some work with Matt Sebold at McMaster University where we showed. Clinicians in practice with lots of experience, a five-second clip of people in our resuscitation rooms.

Jonathan Sherbino: And with that five-second clip, without any other information, they had an accuracy of about two-thirds to predict whether the patient got admitted to the ICU, to the ward, or went home. And so I think first impressions are important in our clinical world. I’m interested to hear what you talked about in our educational world.

Linda Snell: Thank you. Well, my first impression story, I’m always reminded about it when I started off as a resident running the ICU in a small hospital. And one of the nurses came to me and said, you know, you have about one nanosecond before we actually make up our minds about you. And they said, so the nurses are making up their minds about the residents who are rotating through their ICU.

Linda Snell: And then she said, and we often don’t change our opinion on it. So you better make a good first impression. Scared the hell out of me. All right. So first impressions are important. What is a first impression? If we take it from the article, first impressions are also called thin slices or zero acquaintance judgments.

Linda Snell: And these are judgments about people who we don’t know. They’re formed within seconds. And they’re obviously based on incomplete data. And they’re made. Automatically with little consciousness and little awareness. Why is this important? I must say the authors give a really excellent two-page summary of the literature in this area, and I’m going to try and summarize it in five points.

Linda Snell: First impressions are at times predictive of subsequent judgments. For instance, in OSCEs, multiple mini-interviews, or teacher evaluations. First impressions may impact if there is a change in performance. So in other words, if you think somebody’s really good, you may end up overrating them later on or vice versa if your first impression is not so hot.

Linda Snell: But most of the studies on first impressions have been done in medical education in exam contexts and not in real life. Fourth, in competency-based education, first impressions might influence entrustment decisions. The initial entrustment is like a thin slice or a presumptive entrustment, or what could be called a non-acquaintance judgment.

Linda Snell: And finally, there is a thought and some literature suggesting that the act of making a first impression explicit, when you have to actually say what your impression is, influences subsequent ratings compared to situations where the rating is done implicitly. But again, most of these have not been done in the real-life clinical setting.

Linda Snell: So our authors set out to first determine the degree that raters are influenced by first impressions when the learner performance changes in a more naturalistic context. And second, to determine if making a first impression judgment explicit affects subsequent ratings compared to a more natural, which we would think would be an implicit. Rating process. So what do you think? Have I hooked you on Jason Jon Lara?

Jason Frank: I think this topic is a fantastic topic, right? Because it is a important component of all of our assessments, whether they’re structured and high stakes or they’re semi-structured and they’re workplace based or just our informal assessments all day long. So this is a really important topic about that first impression.

Jason Frank: Is there something we need to manage here or is it going to be okay? So you got me at that premise. Premise of this paper is we’re going to move away from the OSCE setting. We’re going to do something a little bit closer to authenticity. And I’ll have more about that later.

Jonathan Sherbino: I think it’s an important topic. I really like the cognitive psychology of so-called first impressions, which is, can I retrieve from long-term memory an exemplar that maps? It’s a pattern recognition task. If the correlation is strong, then yes, I have a diagnosis, clinical diagnosis. But presumably, it’s going to work similar in an educational diagnosis learner.

Jonathan Sherbino: And if there’s not a strong correlation between what I can retrieve, then I might need to use a much more thoughtful, energy intensive, resource intensive type of analysis and work through first principles to understand what’s happening with our learners. But it’s never either or. And so when we get into these either or dichotomies, we get away from the messiness of a real world.

Jonathan Sherbino: Like Jason, I’m going to foreshadow that this is not an authentic type of study. It’S… One step removed from OSCE, but not into the realm of anything that looks authentic. So I think there’s a bit of sleight of hand in how it gets set up, but we’ll leave that for when you describe the methods a bit more.

Lara Varpio: Some of these authors are some of the people I cherish in my world. So when people I adore write, I write, I read. It’s going to happen. Tim Wood and I have been friends and colleagues for many, many years. When I worked back in Ottawa, same with Claire and Deb and Sue. So when they’re writing, I’m reading. Just going to happen. So yeah, I’m in.

Linda Snell: Great. So let’s move on to the methods. And I’m going to summarize the methods for you.

Linda Snell: 2, 2, 6, 3, 1. Okay. That’s the summary.

 Jonathan Sherbino: 42 is the answer. That’s the answer.

Lara Varpio: Dude, that’s like the first time I think I understood what was happening. We agree. The answer, 42. Results. Of course.

Linda Snell: Life, the universe, and everything. Right?

Lara Varpio: Yeah.

Linda Snell: Two Canadian universities, University Of Ottawa and University Of Alberta, so they’ve got East and West. Clinical supervisors from internal medicine, emergency medicine, family medicine, divided into two. Two groups, the implicit and the explicit expression of first impressions, six videos of trainee patient interactions of common clinical problems.

Linda Snell: And I must say, they spent several paragraphs describing how they produced the videos. And I really liked it. It was very well done. You could really understand what they were doing and why they were doing it.

Jonathan Sherbino: Could I just jump in here and say what they do is they staged these cases. These are not filmed cases of residents acting. So it’s not authentic. And that’s what I was referring back to before.

Linda Snell: So what it is, is it’s staged in that they have residents who are sort of given a case and told to act in a certain way, weak, strong or indifferent, basically. And the patient is actually acted by another staff, a staff member.

Linda Snell: Back to the numbers, three experimental conditions of trainee performance, weak to strong, strong to weak, and consistent, the same in either one. So the performance may start off as weak and then improve during the video, or it might be the other way around, or it might stay the same. And they used one rating instrument, a modified mini CEX.

Linda Snell: And for the explicit supervisor group. Remember, there’s two groups of supervisors, explicit and implicit. The explicit one, they stop the video at 60 seconds and say, please provide a first impression global rating.

Linda Snell: And then they go on with the video. And then for both groups, explicit and implicit, not only do they complete the full mini CEX and all the parts of it, but they have a final global rating. So really the only difference in the two groups is…

Linda Snell: The explicit group has one extra step, which is to provide what’s called a first impression, although it really isn’t 60 seconds. The analysis is done with many correlations. I’m not going to go into the details of it. There’s an internal consistency of the six mini CEX item ratings. There’s the item ratings versus the global ratings.

Linda Snell: There’s first impression global ratings versus the regular global rating at the end. And there’s mini-CEX IDA ratings for explicit and implicit. And there’s also between the three conditions, weak to strong, strong to weak, and consistent. And they did calculate a sample size as 25 per group. So comments on the methods, anybody? We’ll go Jason, Jon Lara.

Jason Frank: Okay, this is where I got to declare that I work with all these folks and I adore them all. These are amazing medical educators that I get to play with. I was not involved in this paper. So if I just with fresh eyes reading this paper, I follow the logic. They’re basically saying, yeah, yeah, yeah. We did this with an OSCE before.

Jason Frank: Now let’s do this with some videos that are kind of like what might happen on some wards. Remember that there’s a little caveat here that they had people from multiple specialties and then they had they tried to pick some acute what they called acute problems and said, hey, all of these specialties should be able to rate how this resident tackles this problem.

Jason Frank: That made me pause because. I’m not sure I would look at a resident performance the same as somebody in internal medicine, because I’m looking for different things. I’m looking for, for example, making diagnosis versus maybe definitive management. And I’m wondering how that went with the study, because I didn’t see their checklist and their global rating and how it was worded.

Jason Frank: Nevertheless, the logic’s pretty straightforward. I understand what they did with two examples of each case, make six, and then they had people rate. And then that whole… Does your first impression making it explicit make a difference from the literature? I thought that was clever enough. Good questions.

 Jonathan Sherbino: I don’t have anything of serious critique to offer for the method section. I think it’s a nice, I was going to use the word elegant, but it was a little bit hard to read with all the acronyms and the explicit implicit. And the assumption or the research question, I had to go back to a couple of times.

Jonathan Sherbino: But overall, it’s kind of an elegant little experiment in a simulated fashion. There’s a couple of things I just kind of scratched my head at is why did they unmask their research question? They told everybody, hey, we’re trying to see if first impressions mess you up.

 Jonathan Sherbino: So I wonder if they incepted people to say, oh, go against what you really do or be attentive to what’s happening here. So I don’t understand the purpose of that. The second is they use the mini CEX, which is really old technology. It’s got to be like 25, 30 years old. And apologies to John Norcini if he hears me.

Jonathan Sherbino: It uses a one to nine scale that has unsatisfactory, satisfactory, and superior as the anchors. And if you talk to anybody who knows about scales and measurement, that’s not what we think about as contemporary.

Jonathan Sherbino: And more importantly, their version of the Mini CX has seven questions, which they actually show have a high correlation one to another, which means there’s some massive halo or millstone effect, and you’re just going on the overall global rating anyway. And so they could have got that cognitive burden out of the way and say, what’s your impression at the start?

 Jonathan Sherbino: What’s your impression at the end? Yeah. Because all the other subdomains don’t actually differentiate at all, which tells you that you have something wrong with your scale. But that’s been the problem with scales.

Jonathan Sherbino: That’s been the problem with Mini CX forever. They chose what’s common, but maybe with a little bit more of an eye, they might have thought about using a different scale. It’s a minor point, but I guess worthy of just pointing out to our audience.

Jason Frank: Really good point, Jon.

Lara Varpio: So I have very little to add to that, Linda, because one of the things I do when I read quantitative Papers and I’m trying to figure out how I feel about the methods is I draw this picture. And the goal for me, a good paper is a paper where I can draw this picture without thinking. You know, I write down the research question at the top. I write down the factors that they’re considering and where the intervention happens.

Lara Varpio: It’s like a little flow diagram. And then I look at their analyses. I understand the analyses on the abstract. And I see if that maps back to my picture. And then. If it works, it works. And when I mapped the picture for this study, there was nothing there that I thought was incongruent. So from my perspective, I could draw this. So yeah, works for me.

Linda Snell: Good. Well, the only thing I’ll add to what Jon said is I agree with you, but they didn’t need to use the mini CEX, except that some of the calculations were made after they’d done the mini CEX.

Linda Snell: When they found out that they could just use the global rating. Now, is that new? No, that’s not new. People do know that. So let’s go to the results. And Jon, I have a question for you specifically. The sample size was 25. True, false.

Jonathan Sherbino: Oh, I’m sorry, I didn’t answer my question.

Linda Snell: I just know what the question’s going to be. The sample size was to have 25 per group. They came up with, instead of 50 raters, 45 raters. Does it matter that they did not reach the sample size that they wanted?

Jonathan Sherbino: I guess it’s a philosophical question. And when you’re talking about, you know, less than 10% of the overall sample, I’m not sure if that’s critical, but you do a sample size to say, okay, can we hit a threshold after which we will be able to determine some educationally significant difference?

Jonathan Sherbino: That’s not true. Over which we’ll be able to determine some statistical difference, the difference which you need to subsequently adjudicate to say, does that actually, do you care about it as an educator? And so because they don’t hit their sample size, there is that theoretical risk of a type two error, which says we found no difference.

Jonathan Sherbino: But maybe that’s because we just didn’t have enough people to show the difference. So if they said we need 50 people, we didn’t get 50 people. We found no statistical difference. Then in the back of your mind, you could say, but if you recruited five more people, what would have happened there? And there’s something called a sensitivity analysis where you can simply try to make the people that are missing.

Jonathan Sherbino: And. Put them to the extreme of the outcome and see what happens to the overall analysis. Does if you make five people have some wildly terrible scoring, does that unbalance everything? Was it kind of balanced? And now suddenly you tip it off balance. They didn’t do that type of sensitivity analysis. It’s okay. We’re talking about 10% difference.

Jonathan Sherbino: Eyeballing it, it feels all right. But if you are a hardcore math nerd right now, you’re saying, ah, or whatever they say. 1 0 0 0 1 1 1 0 no no they go ah euclid and then off they go well i was trying to do like a like a computer programming joke which is even more lost on the phd English major right like i think so foreign language gone yeah.

Linda Snell: Let’s get to some of the other results.

Linda Snell: There’s a lot of data, and most of the data reinforces prior studies, which is nice. But what’s really new here? In the explicit group, when the performance was consistent, it started good, they ended good, whatever. The first impression didn’t make a difference. For the other two conditions, good to bad or bad to good, the global rating moved in the expected direction.

Linda Snell: In other words, it suggests that the first impression didn’t make a difference. The raters changed judgment. And there were no significant differences comparing implicit and explicit at the end of Mini-CX global ratings. So the authors summarize this as physician raters adjusted their judgment based on changes in the learner’s performance within a single patient encounter in a workplace context.

Linda Snell: Now, I recognize, Jason, you’re going to say, this is still not a real workplace context, and I agree with you on that. So, in a simulated workplace context. And there was no difference in the global rating as a function of a first impression being made explicitly or formed implicitly.

Linda Snell: So, those are the two relatively new things. In this study. The rest of the stuff just sort of reinforces things. However, there is one statement here that I’m a little uncomfortable with, and I’ll read it to you. It says, it would require a considerable number of participants to achieve statistical significance in these comparisons.

Linda Snell: Therefore, we are comfortable concluding that making a first impression explicit, which is an artificial task, does not influence subsequent ratings. Any more than what would occur when ratings form their first impressions naturally, which would be implicit. This finding supports the generalizability of the results.

Linda Snell: I don’t know what they’re trying to work around there. Maybe one or two of you can comment on that, Jon?

 Jonathan Sherbino: The difference between them that doesn’t achieve statistical significance is so small that you would need to recruit hundreds of people into both groups. And because we’re looking at 25 readers looking at six so-called residents, that’s far more than the feasibility or the real world operational challenges that we have in our system.

Jonathan Sherbino: So we are never going to have observation of practice exceed their totals. And so we probably won’t see some kind of change. But moreover, the question is, right now, we don’t ask people to say, form an explicit judgment at 60 seconds. We probably want to see that having that judgment doesn’t actually affect downstream.

Jonathan Sherbino: So we don’t want to introduce this intervention for any practical reason into our assessment programs. It’s okay to have a snap judgment unconsciously because the results show that things work out okay anyway. The human brain is more complex than what we give it credit for sometimes and we adjust.

Jonathan Sherbino: And that’s what every analysis in this study shows. We anchor and adjust. And that’s how humans get through this complex world. So yeah, I agree with her statement for two reasons. We’d never changed the operationalizing of our system either for the number of raters. And we would never ask people to say, hey, explicitly give your gestalt up front.

Linda Snell: I guess the thing that concerned me was not disagreeing with what you said, Jon, but that sentence, this finding supports the generalizability of the results. It just makes me a little skeptical about this. The discussion I thought was really interesting. However, it describes the two findings and then asks four questions that the authors say they could not explore. It doesn’t really talk about their results.

Linda Snell: It talks about what their results didn’t do or couldn’t do, related to rate or variability, magnitude of change, differs with the valence, self-fulfilling prophecies, and what happens with multiple interactions. All of which are really good questions, but then they say, here, these are some really good questions, but we can’t answer them.

Linda Snell: And the author’s conclusions are that we can apply these findings to, number one, entrustment decisions, because they may be influenced by context and may lead to first impressions being used because you’re entrusting when you don’t have all that much time. And two, and this is maybe a little bit more important, learner handover. If you are, in fact, given an impression of a learner, it may act as a first impression.

Linda Snell: And that’s a whole area that needs exploring. In terms of conclusions, it replicates the past research on the influence of first impressions, but it extends this research, according to the authors, to a workplace-based assessment tool and setting. And I think we’ve already discussed that this is a relatively old-fashioned tool, and it’s not really a workplace setting. So, thoughts on the results, Lara, Jason, Jon?

Lara Varpio: I found the results really quite reassuring. I find it reassuring that if we have a first impression, we can change our first impression. To me, that’s nothing but good news. There’s one piece of this, though, that I find, you know, they talk, the four questions they leave us with saying they couldn’t find them for great.

Lara Varpio: You know what? Every good, if honest to God, if you’re a real researcher in my heart, the more research you do, the more humble you should be because the depth of things you don’t know just becomes more clear. So the fact that they left with four good questions to me is just like, yes, total respect.

Lara Varpio: That’s how I feel about research too. But there’s, I want to add another piece to that, to their list. I want to put a number five or six. I’ve lost track now. Because for me, what I find is, and I think they do note it as a limitation, is that they, you know, we know our assessments can be impacted by implicit bias.

Lara Varpio: So we know that depending on the way the person presents, on all kinds of factors and considerations that individuals are under no control of sort of in their own ways. That we can be biased against individuals. And what I want to know is, can those first impression biases change?

Lara Varpio: Can we make a difference at that moment of first impressions? Can we do something about the bias of assessment? That’s something I would love to know more about. But again, all good research leaves you asking more questions. So that’s a question I would throw down as a possible next step.

 Jonathan Sherbino: Do you mean social bias? Yep. Okay. Yeah. A very, from a cognitive psychology point of view, a very different thing, but equally important, but different kinds of bias, right?

Lara Varpio: Yeah, totally. Yeah, exactly. Different kind of bias, but it’s still there, right? So can we do something about that? Can we, like, are we able to do it? Or is this something that is going to need real structural change? I don’t know.

Jason Frank: These results are super reassuring. So they should make all of us in medical education and health professionals education around the world feel good that at least in these simulated.

Jason Frank: Control conditions with these people who we don’t know a lot about because there’s not a lot of description in the paper about who participated that they were able to change opinions based on what they observe hey that’s great because we’re talking a lot about implicit bias these days that that people have have some things hardwired into their brain from past experiences so this says thoughtful preceptors can change their mind based on what they see that is super helpful We should emphasize this is simulated.

Jason Frank: I’m not sure how generalizable it is to other settings. I’m not sure that this really informs entrustment decisions, which is a different assessment scenario. I’m not sure it informs things beyond a Mini CX and these problems. But nevertheless, it does help me sleep at night when I think about assessment. And I do.

Jonathan Sherbino: I like the results. It confirms a whole… In another setting where the literature has gone, and so shout out to John Brush, who’s at the University Of Virginia. He was the first author on a study that we did that got published in JAMA that looked at this phenomenon. Can you anchor on a place and then adjust to it? So this is a phenomenon that happens all the time.

Jonathan Sherbino: Interestingly, if there’s nothing in the data, the authors point to it. They don’t standardize for how strong to weak versus how weak to strong their cases are. But it seems if you start out good. You go down less than if you start up poor, you don’t go up as high. So where you start from has an impact about what’s the, where you end up.

Jonathan Sherbino: And maybe we have, and certainly from a biological point of view, we have this scarcity mindset. We are wired for fear. We always are looking for the negative. So if we see something negative, it’s hard to us to adjust from that initial anchoring. Their data shows that trend. It doesn’t actually support from an experimental model, that trend, but I think that’s one little. Interesting piece here.

Jason Frank: So don’t be late for a shift. There you go.

Linda Snell: Right. Yeah, exactly. Or for a morning report in my setting. One of you mentioned, we don’t know a lot about the raters. It’s true. The authors didn’t collect demographic data. One of my paperclips would be, it really would be useful to know if there was a difference in any of the usual variables. If somebody who’d been… A supervisor for 20 years was different than somebody who supervised for a year or two, for instance.

Linda Snell: And collecting demographic data is easy. They’re doing a quant study. I would have… Like to have seen that. I also think it would have been interesting in a future study, back to what Lara says about there’s always more questions, to do some sort of a mixed method study to answer some of our why questions, which came up as part of this. But that’s just future research.

Linda Snell: Let’s go to our rating. First of all, on the methods section, we’ll start with the most quantitative person here, Lara, Jason, Jonn.

Lara Varpio: Yeah, the most quantitative person here. I could draw my picture.

Jonathan Sherbino: Did you say constipated?

Lara Varpio: I did not say constipated. Well, you know, it’s been a bang up day.

Lara Varpio: I am hilarious. Honest to God. Anyway, I could draw my picture for this paper. I could figure out the variables. It was all those sorts of things.

Lara Varpio: What do you mean, what? You didn’t get my joke? You didn’t get it? I’ll draw you a picture later.

Linda Snell: That’s agenda number three.

Lara Varpio: On the car ride home. We have a lot of ground to cover, Jason. Just saying. But they didn’t tell me their paradigmatic orientation, which of course is my pet peeve. So I’m going to give it a four.

Jonathan Sherbino: This is, for those of you playing at home.

Jason Frank: Straightforward. Why are you laughing?

Jonathan Sherbino: Jason has missed where he’s supposed to talk in this episode.

Lara Varpio: Almost every time.

Jason Frank: Yeah i’m sorry it’s only when I follow Lava because she’s distracted i don’t know the bingo square like i don’t know the i’ll try to play bingo throw me under the bus for your issue but whatever i’m stuck i’m constipated like that’s awesome term like if you have too many experimental Papers in a row are you constipated i think you are honestly there you go anyway this paper is i like how it walks you through a logic i love the result it’s so reassuring as we said And I’m not sure that it’s as generalizable to real life as the authors have put forward, but it’s one step closer to authenticity in a workplace-based assessment.

Jason Frank: So I’m going to give it a four.

Linda Snell: And there, I must say you mixed up methods and outcomes, but that’s okay.

Jason Frank: It’s true.

Linda Snell: Jon.

Jonathan Sherbino: You’ve heard my critiques. I’m going to give it a four for methods.

Linda Snell: So we’re all into four. Me too. I thought it was very well explained.

Linda Snell: Now, how about impact? We’ll go the reverse. Can you remember which is the reverse, Jason?

Lara Varpio: Why do this to us? Why, Liz? Easy.

 Jonathan Sherbino: You go last, Marlis.

Lara Varpio: I know when I’m going, but Jason doesn’t. Go on, Jonn.

Jonathan Sherbino: It doesn’t matter. He’s still being confused. Big silent pause, Jay. That’s you.

Jonathan Sherbino: I’m going to give this paper a three. It builds on the literature of how readers… Process assessments of trainees. It has an outcome that’s reassuring to the design. The experimentalists and the educational psychologists and the cognitive psychologists will be very pleased of this data.

 Jonathan Sherbino: If you are a program director, you’d be very pleased that you don’t need to do any kind of faculty development to say, hey, keep an open mind because in fact, humans can keep an open mind. It’s how we get through the world. So a three for me.

Linda Snell: Long pause.

Jason Frank: That’s not nice. That’s using a previous bias.

Jason Frank: Oh this is a our first impression i should say this is a three because i’ll use it when we’re if i’m talking to anybody about concerns about first impressions so this is a paper I would pull not johnson i’m just kidding this isn’t another paper i would pull i’m not sure it changes my management with respect to design of programmatic assessment using authentic assessment i’m not sure yet it’s a three you.

Lara Varpio: As someone who doesn’t play in the assessment space very often and those sorts of things, this paper is going to be a solid three because it gives me exactly the argument I need. It’s perfect for helping me to reassure people about immediate biases, first impressions. So yeah, no, three.

Linda Snell: Well, here we are. Our concordance was very good on methods and now it’s very good on impact. I too will give it a three, reassuring, relevant, a little bit of new information and nicely put together.

Jonathan Sherbino: Could I just…

Lara Varpio: Jump in for one second and say both times Lara gave us whole numbers i know you know why i love this so much more i want to give it whoops it was my cause it changed everybody’s cognition when you do the pause like that i got all this i’ve got a whole list of things i got to keep track of to talk about with Jason later so i like gotta keep all that in my brain i’m feeling constipated all right okay.

Linda Snell: For those of you out there in Papers Podcast land You can stop listening to this nonsense and write and tell us what you think. You can write to us at thepaperspodcast at gmail.com.

Linda Snell: Or in our now repaired message link in the website at paperspodcast.com. It’s been great chatting with you three and recognizing that you now recognize when I choose a good paper because you seem to agree with me exactly, both implicitly and explicitly.

Jason Frank: Boo.

Linda Snell: Boo? Is that what you’re saying instead of take care, everybody, now?

Lara Varpio: No, you got to say it like Jason says it.

Linda Snell: Take care, everybody.

Jason Frank: Exactly.

Jonathan Sherbino: That’s a great impersonation. Thanks for listening.

Lara Varpio: You say talk to you later.

Linda Snell: Then Jason, see how it feels.

Jason Frank: Talk to you later.

Linda Snell: It’s not the same. Okay. So bye-bye. Now it’s your turn. Talk to you later.

Jason Frank: No, no.

Jonathan Sherbino: Thanks for listening.

Jonathan Sherbino: Take care, everybody.

Jason Frank: Take care, little buddy.

Lara Varpio: I did not say constipated, did I?

Jason Frank: Yes, you did.

Linda Snell: Yes, you did. Everybody, stop listening. This is deteriorating.

Lara Varpio: It went downhill early, so anyway.

Jason Frank: You’ve been listening to the Papers Podcast. We hope we made you just slightly smarter. The podcast is a production of the Unit For Teaching And Learning at the Karolinska Institute. The executive producer today was my friend Teresa Sorrow.

Jason Frank: The technical producer today was Samuel Lundberg. You can learn more about the Papers Podcast and contact us at www.thepaperspodcast.com. Thank you for listening, everybody, and thank you for all you do. Take care.

Jonathan Sherbino: Welcome back to the Papers Podcast. I think we should go to kind of an abbreviated version. I was thinking the PP cast, but I…

Jason Frank: Let us die.

Jonathan Sherbino: Let us die.

Jason Frank: We could not do that, Jon.

Jonathan Sherbino: No, okay, all right.

Acknowledgment

This transcript was generated using machine transcription technology, followed by manual editing for accuracy and clarity. While we strive for precision, there may be minor discrepancies between the spoken content and the text. We appreciate your understanding and encourage you to refer to the original podcast for the most accurate context.