When an LLM invents a plausible-sounding API that doesn't exist, or cites a paper that was never written, we call it hallucination. The term carries pathological weight—it suggests malfunction, a system seeing things that aren't there. But this framing obscures something important about how these models actually work.
Hallucination isn't a bug in the generative process. It is the generative process. The same mechanism that produces fabricated citations also produces novel code patterns, creative solutions, and useful abstractions. Understanding this changes how we should think about working with these tools.
The Mechanics of Plausibility
An LLM computes a probability distribution over possible next tokens given everything that came before. At each step, it selects from this distribution—sometimes the most likely token, sometimes sampling more broadly. The math is straightforward:
$$P(\text{next token} \mid \text{context}) = \text{softmax}(\text{logits})$$
Notice what's absent from this equation: any notion of truth. The model optimizes for plausibility within the patterns it learned during training, not for correspondence with external reality. A statement can be maximally plausible—high probability given the context—while being completely false.
This isn't a limitation we might engineer away. It's fundamental to how transformer-based language models operate. They're pattern-completion engines trained on text, and text contains both accurate information and convincing-sounding nonsense in roughly equal measure.
The Spectrum of Generation
Consider what happens when you ask a model to write code for a task it hasn't seen before. It can't simply retrieve a solution from memory. Instead, it must compose patterns: syntax from one context, logic from another, API conventions from a third. This composition is generative in the truest sense—it produces something that didn't exist in the training data.
The same mechanism operates across a spectrum:
Interpolation sits at one end. The model produces output that's essentially a recombination of well-attested patterns. When it writes a standard for-loop or a common SQL query, it's interpolating within familiar territory. The risk of error is low because the patterns are heavily reinforced.
Extrapolation occupies the middle ground. The model extends patterns into less familiar combinations. It might suggest an architectural approach that synthesizes ideas from different domains, or propose an API design that follows conventions but addresses a novel use case. Here, the outputs are genuinely creative but also less reliable—the model is reasoning by analogy in territory where its training data provides less direct guidance.
Fabrication sits at the other end. The model produces outputs that have the form of factual claims but no grounding in reality. Invented citations, fictional API methods, nonexistent historical events. These aren't random—they're plausible precisely because the model has learned the patterns of how citations, APIs, and historical claims are typically structured.
Why "Hallucination" Is the Wrong Frame
The pathological framing of hallucination suggests we should try to eliminate it. But elimination isn't possible without eliminating generation itself. A model that never extrapolated beyond its training data would be a lookup table, useful for retrieval but incapable of novel synthesis.
A better framing: hallucination is unlabeled uncertainty. The model doesn't distinguish between interpolation and fabrication in its outputs—it presents both with equal confidence. The problem isn't that the model generates novel content; it's that we can't tell which novel content is grounded and which is invented.
This reframing has practical implications. Instead of asking "how do we stop hallucination?" we should ask "how do we work productively with a system that can't distinguish its reliable outputs from its unreliable ones?"
Confabulation: A More Accurate Analogy
Neuroscience offers a better analogy than hallucination. Confabulation occurs when the brain fills gaps in memory or perception with plausible narratives, without awareness that it's doing so. Patients with certain neurological conditions will confidently describe events that never happened, not because they're lying but because their brains are generating coherent narratives to fill gaps.
LLMs do something structurally similar. When asked about a topic where training data is sparse, the model doesn't return "I don't know." It generates plausible content that fits the patterns of its training. It confabulates.
Here's a concrete example. Ask a model to implement PKCE (Proof Key for Code Exchange) for OAuth:
def generate_code_challenge(verifier):
import hashlib
import base64
digest = hashlib.sha256(verifier.encode()).digest()
return base64.urlsafe_b64encode(digest).rstrip(b'=').decode()
This is correct. But a model might also produce:
def generate_code_challenge(verifier):
import hashlib
import base64
# Plausible but wrong: MD5 isn't used in PKCE
digest = hashlib.md5(verifier.encode()).digest()
return base64.b64encode(digest).decode()
Both follow the same structural pattern: import hashing, import encoding, hash the verifier, encode the result. The second version is wrong in ways that matter for security, but it's plausible—it follows the form of the correct solution. The model isn't broken when it produces the second version. It's doing exactly what it always does: generating plausible completions.
The Productive Uses of Extrapolation
If we accept that extrapolation and fabrication are two faces of the same mechanism, we can start asking how to harness the former while managing the latter.
Design exploration is one productive mode. When you're uncertain what API you want to build, asking a model to sketch possibilities can surface options you hadn't considered. The model will invent methods that don't exist—but sometimes those invented methods represent good ideas.
# A model might generate this for a data versioning library:
class Dataset:
def commit(self, message: str) -> str:
"""Save current state with a message. Returns commit hash."""
...
def checkout(self, ref: str) -> None:
"""Restore dataset to a previous state."""
...
def diff(self, ref: str) -> DatasetDiff:
"""Compare current state to a reference."""
...
This is pure fabrication—no such library exists with this exact API. But the fabrication is useful. It shows what a git-like interface for data versioning might look like. The model has synthesized patterns from version control with patterns from data handling to suggest a coherent design. That's valuable design exploration, not error.
Pattern synthesis is another productive mode. Models can combine ideas from different domains in ways that suggest novel architectures. A model asked to design a state machine hook for React might produce:
function useMachine(config) {
const [state, setState] = useState(config.initial);
const send = useCallback((event) => {
const next = config.states[state]?.on?.[event];
if (next) setState(next);
}, [state, config]);
const can = useCallback((event) => {
return !!config.states[state]?.on?.[event];
}, [state, config]);
return { state, send, can };
}
This might not match any existing library's implementation exactly, but it captures a reasonable approach to the problem. The model has interpolated between React patterns and state machine patterns to produce something coherent. Whether this specific implementation is what you want depends on your requirements—but as a starting point for design thinking, it's valuable.
The Dangerous Modes
The same extrapolation capability that enables design exploration also enables dangerous failure modes. Understanding these helps us work more safely.
Authority fabrication is perhaps the most pernicious. Models readily generate citations, RFC numbers, CVE identifiers, and other appeals to authority. These fabrications are structurally identical to legitimate references—same formatting, plausible-sounding identifiers, consistent style. Without verification, they're indistinguishable from real citations.
# A model might generate:
# "See RFC 9347 for the specification of this handshake..."
# RFC 9347 might not exist, or might be about something else entirely
Security confabulation is particularly dangerous in code generation. The model might produce code that looks secure—it has the form of security-conscious code—while missing critical requirements:
# Plausible but insecure: predictable seeding
import random
import time
random.seed(int(time.time()))
token = ''.join(random.choices('abcdefghijklmnopqrstuvwxyz', k=32))
This follows the pattern of "generate a random token" but fails the actual security requirement. A developer who doesn't recognize the vulnerability might ship this code.
Silent failure patterns emerge when models generate error handling that looks correct but isn't:
try:
result = complex_operation()
except Exception:
pass # Looks like handling, actually hiding failures
The model has learned the pattern "try/except handles errors" without the deeper understanding that silent exception swallowing is usually wrong.
Working With Uncertainty
Given that we can't eliminate hallucination without eliminating generation, how should we work with these systems?
Verification is mandatory for factual claims. Any claim that could be checked—API existence, citation validity, security properties—must be checked. The model's confidence is not evidence of correctness. This is tedious but essential.
Use generation for exploration, not implementation. Models are most valuable when they're expanding your option space—suggesting approaches you might not have considered, sketching possible designs, identifying patterns. They're least reliable when asked to produce production-ready implementations of complex systems.
Temperature as a tool. The temperature parameter controls how much the model samples from lower-probability options. Higher temperatures produce more diverse outputs—more creative, but also more likely to fabricate. Lower temperatures stick closer to high-probability patterns—more reliable, but less novel.
A reasonable workflow uses temperature deliberately:
- High temperature (0.9-1.2) for brainstorming and exploration
- Medium temperature (0.5-0.7) for drafting and synthesis
- Low temperature (0.1-0.3) for implementation and refinement
Treat outputs as drafts, not deliverables. Model outputs should be starting points for human review, not endpoints. This isn't because models are bad at their job—it's because their job is generating plausible text, not generating verified truth.
The Underlying Trade-off
There's a fundamental trade-off in how we might design these systems. At one extreme, a model could refuse to generate anything it wasn't certain about. Such a model would rarely hallucinate—but it would also rarely be useful. It couldn't help with novel problems, couldn't suggest designs, couldn't explore options.
At the other extreme, a model could generate freely without any tendency toward caution. Such a model would be maximally generative—but its outputs would require maximal verification, and it would regularly produce confident-sounding fabrications.
Current models sit somewhere between these extremes, and different use cases call for different positions. Retrieval-augmented generation (RAG) pushes toward the conservative end by grounding outputs in retrieved documents. Fine-tuning on specific domains can improve reliability within those domains. But the fundamental tension remains: generativity and reliability pull in opposite directions.
Implications for Practice
If you're using LLMs in a software development workflow, a few principles follow from this understanding:
Build verification into your process. Don't rely on the model to be right—build systems that catch when it's wrong. Test coverage, type checking, linting, security scanning, code review. These become more important, not less, when generative AI is involved.
Use generation as a complement to expertise, not a replacement. A developer who understands security can use a model to draft code faster, catching the security confabulations because they know what to look for. A developer who doesn't understand security is actively endangered by a model that generates plausible-looking insecure code.
Be specific about what you need. Vague prompts invite extrapolation. If you want the model to stick to documented APIs, say so. If you want it to note uncertainty, request that explicitly. The more constrained the task, the more reliable the output.
Iterate rather than expect perfection. Use the model's output as a first draft, then refine. Ask for alternatives. Request explanations. The model is a tool for accelerating iteration, not a source of ready-made solutions.
Conclusion
Hallucination is the wrong word for what LLMs do. They don't perceive things that aren't there—they generate things that are plausible but unverified. This is sometimes useful (design exploration, pattern synthesis, brainstorming) and sometimes dangerous (fabricated citations, security confabulation, silent failures).
The path forward isn't to eliminate hallucination—that would eliminate generation. It's to understand the mechanism well enough to use it productively: leveraging extrapolation for exploration while building verification systems to catch fabrication.
Models that can distinguish their reliable outputs from their unreliable ones would represent a significant advance. Until then, that distinction is the responsibility of the humans using these tools.