A summary includes a figure that was never in the document. In a list of cases, one does not exist, and a reply to a guest promises a refund the policy does not allow. Each one reads exactly as well as the parts that are right, which is what makes it dangerous. The industry calls it hallucination, and every AI tool your people use does it some of the time.
Why it happens
A language model writes the most likely next words based on everything that came before them. That makes it fluent whether or not it is right. When the model does not have the answer, the most likely-sounding text is often a plausible answer rather than an admission that it does not know.
OpenAI's own researchers published an explanation in 2025: the way models are trained and scored rewards a confident guess over saying "I don't know", much as a multiple-choice test rewards guessing. Although newer models hallucinate less, especially when they work through a question step by step, none of them has stopped.
Where it shows up most
- Specific facts: names, dates, figures, quotations, case citations and web addresses.
- Summaries of long material, where the model merges two points, drops a qualification or supplies a connection the original never made.
- Questions about things outside the material it was given, which it tends to answer anyway.
- Arithmetic that runs across several steps.
- Requests that push for completeness, such as asking for ten examples when only six exist.
What it costs when it gets out
The business owns what its AI says. In 2024 a Canadian tribunal held Air Canada responsible after its website chatbot told a passenger he could claim a bereavement discount after flying, which the airline's policy did not allow. The airline argued the chatbot was a separate entity, and the tribunal rejected that, finding that the airline had not taken reasonable care that it was accurate.
In June 2025 the High Court of England and Wales dealt with fictitious citations put before it in two separate cases, and warned that the people who lead firms will be asked what they did to prevent it. The law firm policy article covers what followed.
What to check
- Check every figure against the source it came from. If you cannot find the source, the figure does not go out.
- Open the original for every name, quotation and citation. A citation that cannot be found is a fabrication until proven otherwise.
- Test every summary by picking two specific claims and finding them in the original. If either is missing or changed, read the whole summary against it.
- Treat any answer to a question the material did not cover with most suspicion, because that is where invention is most likely.
- Click every link a tool offers as its source, and check that it says what the tool claims.
Habits that make it rarer
- Give the tool the material and tell it to answer only from that, and to say so when the answer is not there.
- So that checking takes seconds, ask it to quote the passage each point comes from and say where to find it.
- Keep requests narrow. One question about one document is answered more reliably than a broad question about everything.
- Prefer tools that answer from your own documents and show their sources, such as Copilot working across your files, over a general chat answering from memory.
- Do not ask it for things it cannot know, such as last week's prices or what a client decided in a meeting it never saw.
What it means for the policy
Hallucination is the reason every AI policy has a review rule. Under it, anything with a figure, anything about a named person, anything that becomes a commitment and anything a client will read as a statement of fact gets read by a person first. What an AI policy should contain sets that rule out in full.
Anything that talks to customers directly, such as a booking assistant or a support chat, needs tighter limits still: answers drawn only from approved content and a route to a person whenever it is unsure.
Where this fits
Teaching people where AI goes wrong is part of every level of AI literacy, and of the first month with Copilot. Tracking which outputs people reject, and fixing the cause, is the improving work in Trackside, and measuring against a before is what tells you whether checking has eaten the time the tool saved.