Switching to finance from a different field of study

Hi. So pretty much from the age of 16 to my current age (23), I've been studying all things medicine. Originally, this was chiefly out of parental indoctrination. Then, I fell into the so called mimetic trap. That is to say, I just kept at it mainly because I was and still am surrounded by classmates who also study medicine. And I just didn't want to fall behind. I'm at a point in my life when I get to sit down and think what I actually want to do. And things like investment banking, consulting and financial risk management seem to be right up my street. But I really dread the math part. I know that it will be extremely difficult because I don't have the intuitive grasp of the fundamental knowledge that one has to hone from school. I was just wondering how impactful would this gap be on my financial career as a whole? Should I major in math in order to make up for it and only then proceed to Finance? 4 additional years sound really daunting. Just wondering if anyone has any relevant experience.