How much does college [geographical] prestige actually matter?

Hi,

I have the option of going to a top east coast liberal arts college (Amherst) or a more STEM based liberal arts college on the West Coast (Harvey Mudd).

I know normally if I was deadset on finance I'd choose Amherst, but I don't exactly know what I'd like to do -- my interests range from finance to tech.

I've read that the name of the college matters and that, at least on the east coast, Amherst will be better known than Mudd. However, I graduated from a top prep school (think Andover or Exeter).

In this case, is the college I go to still very valuable, or can I rely on my high school network for alumni connections in the finance world? IMO the education I'll receive at Harvey Mudd is better than Amherst (especially if I decide I wanna do tech), but I don't want to screw myself from ever working on the east coast by going there. Could I hypothetically work at a finance firm's west coast office and transfer to the east coast?

Thanks.