AI Will Back Down When You Push It. That's Not the Same as Being Right.

Hinweis der Redaktion: Diesen Artikel unseres US-amerikanischen Autors Andrew Giordano veröffentlichen wir im englischen Original. Die wichtigsten Aussagen des Artikels haben wir am Ende auf Deutsch zusammengefasst.

I was working through a project with a client recently. Their request was one I'd heard before: get the sales team updating Salesforce more consistently and more accurately. I knew the shape of the problem. What I didn't have was the current technical documentation on the systems I was considering, so I asked an AI platform to pull it and speed up the research. What I got back was confidently wrong, three separate times.

First, it told me a platform couldn't do something I knew from direct experience it could. I pushed back. It folded immediately and admitted the error. Then it warned me that a different approach would produce bad data because it wouldn't respect existing validation rules. That sounded serious, and it also wasn't true. I pushed back again, and it reversed itself again. Then it proposed an architecture. I pushed back a third time, because I knew a better path, and it agreed that my version was actually the stronger fix. Three confident, specific, wrong claims. Three reversals, and only because I happened to know enough to catch each one.

A pattern, not an outlier

This entire process was unnerving, and I started thinking about the organizational risks that come from blindly trusting AI outputs. If I hadn't had the experience to challenge any of those three answers, all three would have shipped straight into the project.

The data suggests this isn't a one-off. KPMG and the University of Melbourne's 2025 global AI trust study (48,000+ people, 47 countries) found that 66% of employees rely on AI output without evaluating its accuracy, and 56% have made work mistakes because of it. And it seems things haven't changed much in 2026. Glean's Work AI Institute, in collaboration with researchers from Stanford, UC Berkeley, and several other universities, published its Work AI Index 2026 in June, based on a survey of 6,000 full-time digital workers across the US, UK, and Australia. It found that 69% of AI users admit to shipping work they haven't verified, don't fully understand, or can't confidently stand behind. The same report notes that workers now spend an average of 6.4 hours a week just making AI output usable: feeding it missing context, checking its claims, and cleaning up the confident-but-wrong answers it leaves behind.

None of that means AI is unusable. It means AI can't yet reliably tell good information from bad information on its own, especially when it's pulling from public sources rather than a system it's been specifically trained on. When it doesn't know an answer, it doesn't always say so. It will find a source from a year ago and state it as current fact, missing an update published three months later. It will take your own idea, hand it back to you, and present it as independent confirmation.

Now picture an employee without deep expertise in the topic, using that same AI platform as their only source of guidance on how to do their job.

What most AI rollouts are missing

Working through engagements like this one, the same three gaps show up again and again.

A source of truth and instructions (a master prompt) that the AI is required to check first before responding to a user's prompt. The master prompt should include product fact references, instructions reflecting access permissions by role, what's off-limits, what counts as current versus stale, and how to respond if the information needs to be externalized. This master prompt has to be continuously evaluated and kept current (including any linked information). The AI has to be configured to never answer without following the master prompt instructions first. This is buildable, and it should be considered non-negotiable.

An actual policy on what employees can and can't do with AI. Which tools are approved, what can never be uploaded, who owns which categories of output, and what an employee is supposed to do the moment something looks wrong or something they were never meant to see.

Real training, with consequences attached. Not a slide deck once a year, but certification, assessment, and a formal disciplinary process if misuse is reported. A policy without a consequence is a suggestion.

None of this works if it sits with a manager three layers down who needs permission to make a call. It needs an owner at the executive table. Anything less, and it becomes the item nobody has the authority to address, allowing the risk it was supposed to catch to grow.

The upfront work to build that structure is lower than the cost of cleaning up after it's skipped. Three confidently wrong answers in one conversation is what it looks like when a system defers to whoever pushes back the hardest, not to whoever is right. Most employees using AI today have no way of knowing the difference.