Would you rather know if somebody was really good at something or really bad?
Yesterday, noon rounds were presented by a guest speaker from the University of Calgary on a topic specific to Medical Education. Dr Ma was presenting her research on assessment tools for clinical competence with procedural skills in internal medicine. She quite eloquently demonstrated the rationale and the problem which she was attempting to find an answer for. I was most impressed at the actual experimental design of her research which allowed her to demonstrate some fascinating results.
One of the particular themes on which she spoke was how many of the assessment tools we have that are check-list oriented are grounded in attempting to assess Competence. That might sound obvious, but what she pointed out was that the majority of them do not have a foolproof method to detect Incompetence. By comparing global assessments of students' performance to the check-list assessments, many experts would deem some of the students who passed according to a check-list to remain incompetent in terms of the particular skill that was being assessed. Many of the errors that were being committed that led an expert to judge a student incompetent were related to major patient safety issues - breaking sterility, taking too many attempts, or failure to prepare effectively. This led them to think that perhaps it would be a far worthier goal to assess incompetence by developing a check-list that enumerates errors that would be unacceptable. Thus, one would pass by not making errors as opposed to demonstrating competence.
At first glance, I like this approach, but I think that it could only be used in certain contexts; if the goal is assessment of competence at an end-of-session examination, you want to know they can do it, not just that they aren't making mistakes. Perhaps the best method would combine these two methods simultaneously - but as Dr Ma pointed out, those experiments haven't been done yet, so we'll just have to wait and see.
As I turned my attention towards assessment in medical education in general, I found myself reflecting on how the choice of evaluation method has changed throughout our four years of medical school. We have often had large scale MCQ exams with occasional short answer sections mixed in. As we progress, we also have OSCE's which are designed to assess our clinical competence in addition to our knowledge. I've often wondered how multiple choice exams can accurately determine knowledge acquisition; I've always found them frustrating because I can't write an explanation as to why I have selected the answer I have, which you can do in short answer questions. However, it turns out, that there is a lot of great evidence that test results from major MCQ examinations are able to predict future clinical competence and performance. In fact, in JAMA in 2007, there was a paper published that demonstrated that test scores can even predict the number of complaints to the college a physician might trigger.
Despite their utility, it is clear that a MCQ exam cannot be used to determine competence in performing a procedure and that performance-based assessments must also be incorporated into the assessment structure of medical education. Direct observation of students with patients or standardized patients seems to be a gold standard for assessment with the prevalence of OSCEs in undergraduate medical education. However, it's important to note that, as Swanson et al point out, despite our best attempts at high fidelity simulation, learners behave differently in these environments than they would in real life.
It is clear that designing assessment tools is not as straightforward as one might think. I'm reminded of my experiences in survey design and how challenging the rigorous methods of validation are that must needs be taken to ensure they are used effectively. I am equally as reminded of my frustration when I had to mark lab reports as a grad student and had to develop my own rubric for evaluation. Dr Ma's presentation was illuminating both in strategies that one could employ to determine the effectiveness of evaluation tools, but also as a reminder that going forward, perhaps I should not always trust the tool I'm being given to evaluate my future students, and that a simple pass/fail approach can sometimes be just enough.
~LG
No comments:
Post a Comment