Took a lifestyle job that's too stressful - advice needed

More than a year ago I made a very conscious choice to switch my job from a large LO to go work for one of the big captive balance sheet investors. It was a new group and I felt like it was the right move, because I found the LO to be too stressful, political and generally too uptight. Also the industry was declining and if I looked up to the senior people that had made it, it was clear I won't do nearly as well even if I worked twice as hard just because of industry pressures. I was looking forward to starting a family, finally find some time to get engaged in hobbies I've been neglecting and spend more time with friends. I was super lucky, because I managed to make that switch in the midst of COVID.

At the beginning it was fantastic - 9-5, colleagues were super nice, way less stressful. Fast forward and close to 1 year in, my workload is piling up and I feel like in my group I'm given the projects that are pretty much mission critical. It contributes to my stress levels, strains the relationships with my junior colleagues that are staffed on with me and obviously impacts my own hours. Basically that the amount of responsibility that I have been given has grown to an unpleasant level. I have direct exposure and must report on progress to top levels of senior management. I know this sounds like a dream to a lot of you that are extremely career oriented, but I just wanted a chill job.. My idea when I joined was that I'd put myself on a slow career progression curve and not raise my hand for much of anything. Instead I'm given the most PITA, complex work that I really don't want to be dealing with and my manager is one of those types that doesn't ask you, he tells. I've been thinking to perhaps plot my exit to another group internally that's more chill (we are a fast growing team with a high profile) in another year or so, but I'm not sure how to engineer this so it doesn't come across as personal (it's a fairly territorial place).

This month I found out we are expecting our first baby, so I feel even more down about the whole situation. I know that a lot of you here are looking for ways to accelerate your career and grow your responsibility. Any advice on how to stall it without totally screwing up your reputation internally?