Job Offer from Friends with Benefits (Conflict of Interest Question)

Hi everyone, so I have been sleeping with a guy on and off for about a year and a half, and we never really made it quite to the dating stage (distance etc.), but as I plan to move back to the area, I've been looking for a job in the financial services sector. His area of work is exactly where I would want to be, and he is one of the top people for hiring decisions there. This would mean that if I were to join his firm, he would be my direct supervisor (well maybe boss's boss, depending). He works at a very reputable and prestigious firm and it has definitely been on my radar, but I've always tried to be careful about talking about his firm because of the obvious conflict of interest. Recently he just offered me a job, and I'm not sure how to take it. I'm qualified (not the greatest candidate due to my different industry experience, but solid all the same) and can certainly make my way through the recruiting process without his help, but I'm just worried about the day-to-day life if I do take a job with the firm. I just finished my degree and we are both very professional, the firm is great and the salary/perks would be solid, but it's a conflict of interest that I'm not sure I want to deal with. Thoughts would be really appreciated!