The illusion of presence

When you talk to a large language model, something unexpected happens: it feels like you’re talking to a person. The conversation flows. It responds to your tone. It seems to understand context. It even picks up on implications and reads between the lines.

This feeling is one of the most dangerous aspects of modern AI — not because the technology is malicious, but because the illusion is so convincing that it distorts how we perceive what we’re actually using.

Why the illusion is so strong

LLMs are trained on enormous amounts of human text — books, articles, conversations, transcripts. They’ve learned, at a statistical level, how humans communicate: the rhythm of dialogue, the patterns of explanation, the subtle ways we adjust tone based on context. When an LLM responds to you, it’s doing exactly what it was trained to do: generate text that looks and sounds human.

This is remarkably effective. It’s so effective that both developers and business decision-makers often slip into thinking there’s agency or understanding behind the responses. The natural temptation is to anthropomorphize — to imagine a “someone” is thinking on the other end of the conversation.

But here’s the reality: there is no “someone.” There is a statistical model producing the most likely next token given its training and your input. It sounds like a person because it was trained on text written by people, not because any person is actually present.

Where this matters most: business decisions

The danger crystallizes when business decisions are at stake. A CEO considering whether to adopt an AI system, a product manager deciding how much to rely on AI recommendations, a finance team wondering if they can use AI output without human review — all of these decisions hinge on accurately understanding what AI is and isn’t.

And here’s where marketing becomes the real problem. The entire AI industry is built on excitement and hype. Vendors market their LLMs as “intelligent,” “autonomous,” “understanding,” “reasoning.” These words carry weight. They carry the implication of human-like agency and judgment. And that implication is, by design, misleading.

The hype serves a business purpose — it attracts investment, drives adoption, sells products. There’s nothing inherently wrong with marketing. But when the hype becomes decoupled from reality, when business leaders start making decisions based on the illusion rather than the mechanism, real problems emerge.

What gets confused

The belief: “This AI thinks through problems the way my team does.”
The reality: It computes probabilities over its training distribution. Modern techniques like chain-of-thought extend this into genuine problem-solving, including on problems it has never seen. But the process is statistical inference, not goal-directed deliberation. It can reach a correct answer through a path no human would take — and miss an obvious one a human would catch immediately.

The belief: “The AI understands my business context.”
The reality: It matches your context against patterns from its training data. When your situation resembles something it has seen before, the output looks sharp. When your situation is genuinely novel, or depends on tacit knowledge that was never written down, the output sounds equally confident and is much less reliable. The model cannot tell you which case it is in.

The belief: “I can use AI output as a primary decision input.”
The reality: It depends on the task. For bounded work with verifiable output — drafting, summarising, code generation — LLMs are reliable. For open-ended or high-stakes decisions, treat the output as a strong first draft. Tooling like retrieval and citations reduces hallucination risk. It does not eliminate it.

The belief: “AI recommendations are objective because they’re based on data and math.”
The reality: Every model reflects two layers of human choice: the training data, and the behaviours reinforced during post-training. The model has a perspective. Understand that perspective before you rely on it.

The path forward

This isn’t an argument against using LLMs. They’re useful tools. But utility requires clarity.

First, understand that you’re not talking to a thinking agent. You’re querying a pattern-matching engine. That’s not insulting to the tool; it’s just honest.

Second, when evaluating AI systems for your business, look past the marketing language. Ask specific questions: What was it trained on? How does it fail? What’s the mechanism, not the promise? Has it been independently tested? What do humans need to verify?

Third, resist the seductive illusion. The fact that an LLM sounds intelligent doesn’t mean it is intelligent in any meaningful sense. The fact that it passes a Turing test (sort of) doesn’t mean there’s understanding happening.

Your role as a decision-maker is to see the mechanism clearly, acknowledge what the hype is selling, and make choices based on what the technology actually does — not what it feels like when you’re talking to it.

If you’re evaluating an AI system for your business and want to see past the marketing, that’s the kind of problem I help with. Get in touch.