What is a hallucination?
A hallucination is output from a generative AI model that reads as fluent and confident but is false. The model may invent a court case, misstate a contract term, produce a plausible but wrong part number, or report that it updated a record when it did not. The text looks exactly like a correct answer, which is what makes the failure dangerous.
Hallucination is a property of how large language models work. The model predicts the next token from patterns in its training data. It has no internal table of facts to check against and no built-in sense of when it is guessing. When the prompt asks for something the model does not know, the most probable continuation is often a well-formed fabrication.
Why hallucination matters
Lawyers have been sanctioned for filing briefs that cited cases a chatbot invented. A collections representative who quotes a balance the model made up creates a compliance problem and a dispute. A maintenance assistant that invents a torque specification can damage equipment or injure someone. In every case the user trusted output that looked authoritative.
The business consequence is that generative AI cannot be deployed as an oracle. It must be deployed inside a process that supplies it with the right facts, checks what it produces, and keeps a person responsible for anything consequential.
Types of hallucination
| Type | Example | Primary control |
|---|---|---|
| Fabricated fact | Stating a warranty period the policy does not contain | Grounding in retrieved documents |
| Fabricated source | Citing a regulation section or case that does not exist | Citations that link to retrieved text, verified in code |
| Wrong number | Changing a balance or quantity while summarizing | Numbers pulled from the system of record rather than from generated text |
| Unfaithful summary | Adding a conclusion the source document never reached | Faithfulness checks against the source |
| False action report | An agent says it sent the email but the tool call failed | Tool results verified in code and every call logged |
Hallucination in practice
A collections agency deploys an assistant that answers representatives’ questions about state-specific contact rules. In the first test, the model confidently described a call-time restriction that did not exist in the state in question. The team rebuilt the assistant with retrieval-augmented generation (RAG) over the agency’s compliance manual, required every answer to quote the manual passage it relied on, and instructed the model to say when the manual did not cover a question. Hallucinations on the test set dropped to a level the compliance officer accepted, and representatives learned to read the quoted passage rather than the paraphrase.
In Kirk Tech Solutions engagements, the standard pattern is grounding plus a human approval step wherever an agent can change a system of record, so that a hallucinated value is caught before it is written.
What to watch for
- Lower sampling temperature reduces variability but does not stop hallucination, because the most probable answer can still be wrong.
- Larger or newer models hallucinate less often but still hallucinate, so controls belong in the process rather than in model choice alone.
- Ask vendors how their system behaves when the answer is not in the available documents, and test that case yourself.
- Measure hallucination with a fixed test set of questions with known answers, and re-measure after every model or prompt change.
- Treat an agent’s claim that it completed an action as unverified until the application confirms it from the tool’s actual response.