Important Information About Your CFA® Exam Results - Calculating the MPS
Well monkeys, it sure was fun not worrying about the CFA® exam for a couple weeks. But, I'm sure that like me, you all got that special email last night telling everyone to restart their freak out until July 23rd. Thanks, CFA® Institute, that's just what my summer needs.
I remember last year, after finishing level 1, I freaked out fairly continuously until the date the results came out. Primarily, I was concerned with the question, "so, how the hell do they actually grade these things?" I wouldn't suggest this particular course. Luckily, I did learn quite a bit about how the grading process works and how the MPS is decided upon.
Hopefully everyone finds this illuminating or if you're inclined to freak out over the eventual MPS, that this eases your anxiety.
Let me start off with an observation. The first things the email states is to make sure that the email is going to the correct email address. Okay, but if it went to the wrong email address, how would I have gotten the email? Just sayin'. But I digress.
As I'm sure many of you have figured out, the CFA® does have a document online that describes how the grading process works. But, I'm also sure that many of you found it wanting, and this is for good reason, the CFA® is generally pretty hush-hush about their grading process, and the provided documentation can be a little cryptic if you're not familiar with what they're saying. The key method that is utilized is mentioned on page 10, right at the beginning of the "Standard Setting" subsection, which reads:
[The] methodology for arriving at the minimum passing score (MPS)—the modified Angoff Standard Setting Method
This is only mentioned once, and yet is the most important fact in the entire section. So, the real question is, what the hell is the Angoff Method? Knowing how this process works is key to understanding how the CFA® comes up with the MPS and why it changes from year to year. Here's the basic process for standard setting via the Angoff Method:
- Assemble Subject Matter Experts: The Angoff Method requires the input of subject matter experts in order to set a given standard. Some key points from the CFA® demonstrate the first step of the process:
Another quality control during this period is the investigation of all comments and complaints related to the examinations.
Any of you remember filling out a survey or being asked about your opinion on a given question? No, of course not, we're just candidates, which means that the comments and complaints are being provided by their subject matter experts during the first step of the Angoff Method.
- Round One of Grading: The Angoff Method's premise is that with a sufficient number subject matter experts, a uniform decision on what makes a minimally competent candidate can be discerned with enough iterations of "grading rounds." The rounds change slightly from one to the next, and based on what was mentioned previously, it seems likely that the first round of grading is to figure out which questions fall into one of these categories:
Special focus is given to reviews under any of the following circumstances: multiple similar complaints, exam results suggesting more than one correct answer, or exam results that otherwise suggest that a question was confusing or unfair.
If I had to guess, I'd say that this step is what makes the CFA® method a "modified Angoff Method" as opposed to simply an "Angoff Method". Don't quote me, this is just a guess. But, good news everyone, if you thought there were some questions on the exam that were totally unfair, don't worry, the CFA® states explicitly:
If, after investigation, a question is determined to be confusing or unfair, results are adjusted to credit all answers. If more than one answer was correct, then all correct answers are credited.
- Round Two of Grading: After round one, we're now working within the traditional methodology of the Angoff Method. This is noted by the CFA®, which states:
Each participant reviews the entire examination, question by question, and makes an independent judgment on the expected performance of a just-competent candidate on each question on the examination.
What this means in practice is that each subject matter expert takes each question and makes an independent judgment as to the probability of a minimally competent candidate getting a question correct. The data is then collected and compiled and averages are calculated for each question, resulting in a probability for each and every question found on the exam. For example, if every question is assigned a probability of 70%, meaning that a minimally competent candidate has a 70% chance of getting the question correct, then the passing score recommendation for the round is 70%. Obviously, it's a little more varied than that, but the process is the same.
- Round Three, Four, Five,... , Whatever of Grading: Herein lies the bulk of the process found within the Angoff Method. After each round of grading, new information is provided to the subject matter experts, and the process begins anew utilizing the new information. The most common type of information that is provided to subject matter experts is what is known as "impact data". The CFA® explicitly states that they do, in fact, provide impact data to those doing the grading:
Participants review the entire examination a second time after reviewing general impact data and overall actual candidate performance on the exam.
But, of course, the CFA® is a little coy with the details about what "general impact data" actually represents. Typically, impact data is the percentage of candidates who would pass based on the previous round's median or average passing score recommendation. So, if in the previous round, the score recommendation ended up being, let's say, 65%, the impact data associated with this recommendation would be the percentage of candidates who would have passed had the final recommendation been 65%. Or, put more simply, in the above quote, replace the word "and" with "which is" (I have no idea why they wrote it the way they did). This process may be repeated as many times as necessary to arrive at a consensus recommendation. Once a consensus is reached, the recommendation is sent up to the CFA® Institute Board of Governors who then sets the MPS based on this recommendation. If I had to guess, I doubt the board significantly changes the recommendation, if at all.
From a high level, this is how the CFA® exams are graded. So, if someone tries to tell you that the MPS is 70% of the top 1% of scores, they're wrong. Maybe that's how they used to do it, but it isn't done that way anymore. Also, this should clear up why it takes so damn long to grade the exams!
Now that everyone is armed with the knowledge of how their exam is being graded, I hope we can all return to not caring about the CFA® results until July 23rd!
CFA® Institute does not endorse, promote or warrant the accuracy or quality of Wall Street Oasis. CFA® and Chartered Financial Analyst® are registered trademarks owned by CFA® Institute.
Thanks for the great post! I plan on taking CFA L1 next summer and am already nervous at the thought of it.
Great post. Lots of good info in there-- thanks for putting in the time and quashing a few rumors. I took Level 2 in June. It's tough to know whether to look forward to July 23 or hope it never comes. I think most, if not all, of the 8 full PTs I took were harder than the real thing, but I historically did not gauge how well I did on any.
Thanks for the info.
I a-m s-o r-e-a-d-y f-o-r l-i-f-e a-f-t-e-r c-f-a
I hate to break it to you, but it's pretty anticlimactic. To be honest, sometimes I miss the pain. It's nice to have memorial days back though.
After taking level 2 on 6/1 I've had the best 2 weeks. The feeling of going home after work and flipping on the xbox or binge watching that one show you've always wanted to get into or just sleeping.....without that dreaded thought that always comes along and ruins everything before the exam...
Enim eveniet architecto et ut autem vero. Ipsum dicta iste nesciunt qui. Repellat odio esse neque adipisci velit repellat cupiditate. Dolores dolor totam veritatis fugit est dolores. Molestias praesentium possimus nulla nulla sequi aspernatur ex eveniet.
See All Comments - 100% Free
WSO depends on everyone being able to pitch in when they know something. Unlock with your email and get bonus: 6 financial modeling lessons free ($199 value)
or Unlock with your social account...