Why Does AI Lie So Confidently? The Anatomy of Hallucination

In 2023, a lawyer in New York cited six court decisions as precedent in a filing. They had names, dates, even quotations. There was just one problem: none of them were real. The lawyer had asked ChatGPT to do the research, and when he asked whether the cases were real, it answered "yes". The court fined the lawyers $5,000.

In AI, this phenomenon has a name: hallucination. The model presents information that isn't true with the same fluency and confidence as real information. Every company bringing AI into its processes needs to understand this risk.

In short:

• Hallucination is when an AI model produces untrue information in a confident voice.
• The reason is simple: language models produce not "the truth" but "the most likely continuation". Guessing when unsure has often been rewarded more than saying "I don't know".
• A tribunal held a company responsible for wrong information its chatbot gave.
• It can't be eliminated entirely, but it can be greatly reduced with document-grounded answers, citations and human review.

What has it cost in real life?

Fake court decisions

The case above went on record in the US as Mata v. Avianca. The judge stressed that the lawyers not only submitted fake citations but didn't promptly admit the error when questioned. After this case, courts in many countries began introducing rules for AI-assisted filings, and similar incidents kept surfacing in the legal world.

An airline's chatbot

In 2022, a Canadian passenger asked the chatbot on Air Canada's website about bereavement fares after his grandmother died. The bot said he could buy a full-price ticket and apply for the discount within 90 days. The airline's actual policy didn't allow the discount to be applied after travel.

When the passenger complained, the airline argued that the chatbot was "a separate legal entity that is responsible for its own actions". British Columbia's Civil Resolution Tribunal rejected that argument in its 2024 decision: the company was responsible for all information on its website, including what its chatbot said. The airline was ordered to pay the difference.

The amount was small, but the precedent was big: what your chatbot says, you say.

One sentence in a launch

In February 2023, when Google introduced its AI chatbot Bard, the demo it shared had the bot give a wrong fact about the James Webb Space Telescope: that it took the first picture of a planet outside our solar system. Astronomers spotted the error immediately. Alphabet shares fell sharply after the news, wiping about $100 billion off the company's market value.

Why does hallucination happen?

The model produces the "likely", not the "true"

Large language models are trained on huge amounts of text to learn one thing: predicting the most likely next word in a text. That ability is astonishingly powerful. It captures grammar, style, reasoning patterns and a great deal of knowledge about the world.

But there's no "database of facts" inside the model. When asked to continue a sentence like "The following decisions were cited as precedent in Mata v. Avianca", the model knows very well what real court decisions look like: the name format, court name, year, page number. If it doesn't recall a real decision, it can produce one that looks real, because what it learned in training was to produce realistic-looking text.

Guessing has been rewarded

OpenAI's 2025 research paper "Why Language Models Hallucinate" makes an interesting point: model training and evaluation often reward guessing over acknowledging uncertainty.

Picture a multiple-choice exam. Leave a question you don't know blank and you surely get zero; guess and you have a chance. When wrong answers carry no penalty, the smart strategy is always to guess. According to the paper, evaluations that measure models only by accuracy teach them exactly that. The researchers' proposal: evaluation methods that penalize confident errors more than saying "I don't know".

When is it more common?

• Rare information: little-known people, small companies, local events. The model saw few examples of these in training.
• Exact values: dates, numbers, source names, URLs, articles of law.
• Events after the training cutoff: the model can't know what happened after its knowledge cutoff, but may try to produce an answer when asked.
• Leading questions: questions with a false premise, like "Explain why X won award Y."

How do you reduce hallucination?

The good news: current models hallucinate much less than earlier generations and are more willing to say when they're unsure. But the risk isn't zero. Especially in business processes, these methods make a difference.

1. Ground the model in documents

Instead of letting the model answer "from memory", have it base answers on documents you provide. The technical name for this is RAG (Retrieval-Augmented Generation): documents relevant to the user's question are first retrieved from your company's own sources, then the model is told to answer based on them.

In the Air Canada case, grounding the chatbot's answer in the company's actual refund policy document would very likely have prevented the error.

2. Let it say "I don't know"

State it explicitly in your prompt: "If the answer isn't in the provided documents, say you don't know. Don't guess." This simple instruction noticeably reduces the model's tendency to fill gaps with fabrications. For more techniques, see our Claude prompting guide.

3. Ask for sources and quotes

Ask the model to show which part of the document supports each claim in its answer, and to retract any claim it can't back with a quote. Requiring citations both reduces hallucination and makes checking easier.

4. Verify critical information

Legal citations, medical information, financial figures, articles of law and commitments made to customers should always be checked by a person against a primary source. The New York lawyer's mistake wasn't using AI; it was not verifying the output.

5. Draw boundaries in customer-facing systems

If you're building an AI assistant that talks to customers:

• Tie answers on binding topics like prices, refunds and warranties to approved documents.
• Define topics where the model can't make commitments on its own, and have it hand those to a human.
• Log conversations and regularly review a sample.

Frequently asked questions

Is hallucination a "bug" or the model's nature?

Both. Hallucination is a natural by-product of how language models work, but research shows it can be greatly reduced through training and evaluation methods. Indeed, rates keep falling with each new generation of models.

Which AI model doesn't hallucinate?

None. All large language models can hallucinate under some conditions. The difference between models is how often it happens and how honestly the model expresses uncertainty.

If the model can search the web, does it stop hallucinating?

Search greatly reduces the risk, especially for current information, but doesn't remove it. The model can misread a source, rely on an unreliable one or add a detail the source doesn't contain. Checking the source links still matters.

Is using AI in my company risky because of this?

Not when it's designed well. For drafting, summarizing, brainstorming and document-grounded Q&A, AI brings big productivity gains. The risk lies in output going unchecked to customers, courts or official documents. The answer isn't avoiding AI; it's using it within the right boundaries.

AI is like a very fast assistant who speaks confidently but is sometimes wrong. If you want reliable, auditable AI solutions grounded in your company's own documents, reach us through our enterprise software development page. For another security risk with AI agents, see our prompt injection post.

Sources