Got caught with a wrong number in an IC meeting because I trusted AI output.
This is mildly embarrassing to post but figured someone else has been here.
Few months back we were looking at a platform acquisition, mid-size business services company. Decent deal, moving fast, partner was excited about it. I used AI to pull and organize the financials from the CIM because we were under time pressure and I'd done it enough times that I was comfortable with the output. Ran the numbers through, built out the quick model, everything looked clean.
Get into the IC prep call and the partner starts stress testing the EBITDA build. I'm walking him through the add-backs and he stops me on one line. The AI had pulled adjusted EBITDA from one section of the CIM but the actual reconciliation table two pages later told a different story. Different treatment of a one-time item. Not a huge difference in absolute terms but enough to move the margin assumption by about 150 bps, which on a 6x revenue business at the size we were looking at is not nothing.
Wasn't catastrophic. We caught it before anything went out. But it was a bad moment. Partner didn't say much but the look was enough. Spent the next hour manually going line by line through every figure in the model while he waited.
The frustrating part is I knew AI extraction wasn't perfect. I'd seen it get things wrong before on smaller stuff. I just got comfortable because it had been right enough times in a row that I stopped treating it like something that needed checking. That's on me.
Curious how common this actually is because I can't imagine I'm the only one. And for people who've been in a similar spot, did it change how you use these tools or did you just get more careful and move on?
Sam Altman and other AI spokesman have given numerous interviews talking about why hallucinating in LLMs is unavoidable because of how they function at a fundamental level... there are dozens of stories of numerous top consulting and law firms being caught submitting AI slop references to things that don't exist in proposals, research papers, and legal briefs/submissions to customers and courts. It's probably the single most talked about issue brought up when applying these to any research or mathematical output. And you decided to just use blindly use & trust the tool and not verify any of the outputs, something you generally shouldn't even do with a human analyst? Glad it wasn't catastrophic, but you're lucky you didn't get fired.
AI tools are good for automation of existing workflows that run from a fixed template where the output is essentially binary - think of it like a series of if() functions based on static inputs - but if you're trying to use it for a final draft of anything variable that forces it to reference its own decision-making then you're going to risk getting some slop in the mix most of the time. They're good for getting a draft started, not finalizing any sort of important work IMO.
Edit: I missed this - you were putting confidential materials into an AI tool, so I really hope it's a tool your firm pays for/that you have approval to do that with. I'm seeing some NDAs these days that explicitly say you cannot feed their materials into consumer LLM/AI tools without express consent, not to mention a lot more firms are being strict about feeding proprietary/confidential data into any tools not expressly purchased and air gapped by them for security reasons.
Fired? Seriously dude? No one is getting fired over a mistake like this. Yes, if it was a repeated error, maybe eventually, but let’s not be dramatic. The dude was trying to leverage a tool that undoubtedly his firm is paying a lot of money for and has most likely repeatedly told him to use to save time. There are going to be things like this that happen. This isn’t life or death.
I'm just giving my perspective. Objectively, you're correct, what happened in THIS instance is not a big deal. But he is LUCKY that's the case. He is LUCKY that the hallucination by the AI was on something that, while somewhat important, did not create a "real" problem. He is LUCKY that it was caught during a prep call and not in the middle of an IC presentation in front of all the head honchos (thereby embarrassing himself and all the seniors involved) or god forbid somehow getting missed going through that then being caught later (imagine a deal getting done, underperforming, and then upon review they found some important info was hallucinated by some ASO - you think they're not going to look for someone to blame the underperformance on?). He is LUCKY that the partner doesn't seem to want to make an issue out of it.
On the point of embarrassment - that is a real risk. PE is politics, and when you cause a partner to lose face/look bad in front of the rest of the firm it can pose a very real risk to your own longevity at the fund. It is not uncommon (nor is it justified, but it is what it is) for a partner/MD to cut loose a junior who fucked up and made them lose face. Especially when it's over something that they should have known to be careful about. It's been known for years now that AI hallucinates, so not checking the outputs is a big no no.
There's also no context provided that what tool he was using was an air gapped, enterprise product purchased by his fund. If he were just using a personal account with Claude/Grok/GPT to rip through a CIM, that could be a violation of his own fund's cybersec/compliance guidelines and of the NDA signed to obtain said materials. This stuff isn't a joke - there are people at large funds who have been seriously reprimanded or fired for doing stuff like this without making sure the proper channels were cleared because liability in these situations is a very real risk.
I'm 100% overselling how bad this is now, because of how bad it COULD have been - not just for OP but for any other juniors making use of these tools and not carefully reviewing the work. Maybe OP's working at a super chill LMM fund where there's going to be no long-term repercussions and people are chill about an honest mistake. I truly hope that's the case. But larger MM, UMM, and MFs with institutional processes and strict CI handling policies? They are looking for reasons to label underperformance and identify legal risks. Is saving an hour or two of work really worth risking having that label placed on you and risking your job? I personally don't think so, and as much as I love these tools and think there's a ton of use cases for them, there are also some very strict handling procedures I believe everyone needs to adopt before jumping in head first.
Dude if you’re an ASO working with a partner literally he expects half of it to be error riddled
"but the actual reconciliation table two pages later told a different story. Different treatment of a one-time item. Not a huge difference in absolute terms but enough to move the"
this sentence reads like AI slop man.. no character development?
This is how I write all my emails to LPs. This physician roll up is being marked down to cost because the federal reserve told a different story
100% AI generated according to GPTZero btw
Hilarious I can’t wait to hear more of these
Honestly entire deals get done on models with plenty of errors in the pipes so would not let this stress you out
One of my MDs once told me that it's likely every deck has an error, but we should still try our best to ensure there is no errors. Just kind of nature of the business.
I don't know dude. Personally, as a partner I wouldnt be so lenient. You are paid the big bucks for your age because you are supposed to be reliable, good with numbers and hardworking. This shows bad judgment and for me this could taint all your work output. If I want an AI I will use an AI and I will pay 100 a month, not whatever you are paid.
If I cannot rely on my associates to provide error free work in order to make calls which have millions of dollar impact, there is an issue.
Edit: but maybe I am just some old schood mofo
My VP put together an IC memo without me while I was on vacation and completely made up a line item in the financials because he trusted AI.
It’s not that the number was wrong it was that he didn’t know what EBPP stands for so he used AI and it said “earnings before purchase price allocation.”
I saw it later and decided that makes no sense let me figure it out so I thumbed through the model and it’s actually employee bonuses.
Nobody caught him but an IC member did ask about employee bonuses and why we have no info on it.
Reiciendis dignissimos ut est quis vero laudantium qui consectetur. Iure doloremque quis aut et quibusdam eligendi sit. Assumenda magni illo et quas odio distinctio. Soluta quis at itaque dolore cumque adipisci tenetur quo. Qui fuga tempore non temporibus modi occaecati est necessitatibus. Recusandae eius numquam voluptatibus hic iure enim. Sint laborum minima qui laborum et pariatur.
Necessitatibus ea aut excepturi qui. Quia voluptate vel laborum qui nobis vitae sapiente. Enim nihil soluta possimus nostrum. Minima eum ut debitis earum. Nam repudiandae nemo explicabo. Quia quis et neque sit quasi omnis.
Facere placeat veniam soluta deleniti neque. Nam iusto est qui. Sed enim ut nihil expedita nostrum.
See All Comments - 100% Free
WSO depends on everyone being able to pitch in when they know something. Unlock with your email and get bonus: 6 financial modeling lessons free ($199 value)
or Unlock with your social account...