Kevin Eva
How well you fare on a subjective evaluation – whether it’s treating a patient, auditioning for a play or even interviewing for a job – may ride largely on the person who was examined just before you. And the worse the person before you did, the better for you.
That sobering conclusion by University of British Columbia professor Kevin Eva resulted from a controlled experiment that examined whether physician-educators’ grading of younger doctors could be influenced by the quality of those they had previously evaluated.
The study was one of two articles by Dr. Eva, published today in JAMA (the Journal of the American Medical Association), that examined medical school admission and evaluation practices.
In admissions, Dr. Eva and collaborators examined the validity of multiple mini-interviews (MMIs) – a dozen, brief encounters in which applicants are asked to solve a problem through discussion with an interviewer, or through interaction with an actor or another applicant. MMIs have come to replace longer, more traditional interviews at most Canadian medical schools (including UBC’s Faculty of Medicine).
The researchers found that students accepted to McMaster University, the first Canadian medical school to adopt MMIs, performed better years later on licensing examinations than students whom McMaster rejected but who were accepted at other, non-MMI schools.
The evaluation study, conducted with researchers at the University of Manchester, asked physician educators in England and Wales to grade videos of young doctors in their first year of post-graduate training. The first-year doctors portrayed in the videos followed scripts to represent three levels of performances – good, poor, and borderline – as they interviewed and examined actors portraying patients.
Some physician-educators were “primed” by viewing the good performances while others viewed the poor performances. Then both groups were compared on their grading of the borderline performances.
Educators who had been primed by poor performances consistently gave better grades than those who had just watched the good performances – depending on the patient case, the grades were 30 per cent to 100 per cent better.
“This experiment shows that judging someone’s performance – whether it’s clinical skills, essay-writing, or figure skating – is likely to be relative, and that we can’t assume that examiners are working from a fixed, absolute standard, as is expected in current models of education,” says Dr. Eva, the Director of Educational Research and Scholarship in the Department of Medicine and a Senior Scientist at UBC’s Centre for Health Education Scholarship.
“While such assessments are unavoidable in determining whether a student has mastered the required competencies in a field like medicine, we need to take steps to minimize contrast bias – perhaps by continually mixing up the order of the people being examined,” Dr. Eva says. “This is one of the reasons it is important to ensure that there are a sufficient number of evaluations – the more data points, the more reliable the aggregate score will be.”